Avoid probing docs for non-latest versions (
0976443)
The frontend now checks a package’s latest unyanked version before requesting docs, so pinned versioned doc URLs no longer trigger guaranteed-failing API calls. That should cut a large amount of wasted crawler traffic while preserving redirects for docs and rendering version pages without docs when needed.
Other misc changes
- Added a
robots.txtrule to disallow crawling versioned doc paths.