Sphinx linkcheck and broken/redirect occurrences in Python Docs

[rffontenelle]

Running make linkcheck in Doc folder outputs thousands of broken and redirected status. It would be nice to clean this up so we can link-check Python in CI/CD for each commit, but right now it is too polluted.

Some of these occurrences could/should be fixed in the docs itself, others can benefit from Sphinx's linkcheck configs e.g. linkcheck_ignore and linkcheck_allow_redirect.

See linkcheck-log.txt for the full log as of commit f2b7ecb.

Fun stats obtained from the above file:

• Of its 8327 lines, where 7824 are related to BPO -> GH issues. Of 7824, 5744 lines are from whatsnew/changelog and only 20 are not from whatsnew/
• 241 lines are redirection of CPython CVS URL, fixing /tree/ to /blob/ in GitHub URL
• 28 lines are broken links, where one broken link is 'https://' from an example in whatsnet/changelog

The way I see, this steps divide in:

• Clean BPO to GH redirection messages
• Fix broken links
• Clean CPython CVS URL redirections
• Clean GH issues to GH PR redirections
• Fix the rest of the occurrences.

linkcheck_allow_redirect and linkcheck_ignore can be very handy in this case. linkcheck_allow_redirect makes 'ok' status a redirect that is being spotted by linkcheck, and we have linkcheck_ignore as the last resource.

Questions I have before implementing the solution:

• Documentation hosted by Read The Docs may have language enabled so example.com is redirected to example.com/en/latest. To handle occurrences, I could add them to linkcheck_allow_redirect or we can use sphinx-ext-intersphinx to map a keyword to the documentation URL (e.g 'rtd' for read-the-docs docs). The last option allows to map to proper language of the target URL linked, similar on how Weblate did)

• Is there any restrictions to fix broken/redirect links in old whatsnew/.rst?
• Is there any restrictions to fix broken/redirect links in old whatsnew/changelog.rst (i.e. Misc/NEWS)?
• Should I create a single Pull Request for all the fixes?

Linked PRs

• GH-103484: Docs: add linkcheck allowed redirects entries for most cases #103569
• [3.11] GH-103484: Docs: add linkcheck allowed redirects entries for most cases (GH-103569) #103602
• GH-103484: Fix broken links reported by linkcheck #103608
• [3.11] GH-103484: Fix broken links reported by linkcheck (GH-103608) #103683
• GH-103484: Fix redirected permanently URLs #104001
• [3.11] GH-103484: Fix redirected permanently URLs (GH-104001) #104088
• GH-103484: Tell linkcheck to ignore debian manpage redirects #123019
• [3.12] GH-103484: Tell linkcheck to ignore debian manpage redirects (GH-123019) #124136
• [3.13] GH-103484: Tell linkcheck to ignore debian manpage redirects (GH-123019) #124137
• GH-103484: Fix permanently redirects reported by linkcheck #124144
• [3.13] GH-103484: Fix permanently redirects reported by linkcheck (GH-124144) #124151
• [3.12] GH-103484: Fix permanently redirects reported by linkcheck (GH-124144) #124152
• GH-103484: Fix broken links reported by linkcheck #124169
• [3.13] GH-103484: Fix broken links reported by linkcheck (GH-124169) #124179
• [3.12] GH-103484: Fix broken links reported by linkcheck (GH-124169) #124180
• [3.12]: gh-103484: Relink _xxsubinterpretersmodule.c in whatsnew/3.12.rst #124183

[merwok]

What about ignoring the 7824 bpo redirects? They work fine, we control bpo and can keep the redirector working, we don’t need to change 8000 lines in historical whatsnew docs.

The CVS redirections can be fixed, as well as the broken links.

[hugovk]

(For reference, some earlier discussion at https://discuss.python.org/t/sphinx-linkcheck-and-broken-redirect-occurrences-in-python-docs/25687)

• Of its 8327 lines, where 7824 are related to BPO -> GH issues. Of 7824, 5744 lines are from whatsnew/changelog and only 20 are not from whatsnew/

Yes, let's ignore the BPO redirects. There's so many, I think either linkcheck_allow_redirect or linkcheck_ignore, whichever is quicker for linkcheck to process.

We actually have a linkcheck_ignore for BPO, but the regex doesn't match, I guess it's gone out of sync:

cpython/Doc/conf.py

Line 258 in 7f3c106

linkcheck_ignore = [r'https://bugs.python.org/(issue)?\d+']

• 241 lines are redirection of CPython CVS URL, fixing /tree/ to /blob/ in GitHub URL

Please could you give some examples of these?

• Documentation hosted by Read The Docs may have language enabled so example.com is redirected to example.com/en/latest. To handle occurrences, I could add them to linkcheck_allow_redirect or we can use sphinx-ext-intersphinx to map a keyword to the documentation URL (e.g 'rtd' for read-the-docs docs). The last option allows to map to proper language of the target URL linked, similar on how Weblate did)

If there's not too many, we could add to linkcheck_allow_redirect, similar to the devguide. There may be some other useful rules to copy over:

https://github.com/python/devguide/blob/79e1a1f2961ba0882d5f0274f7c8876e36ca5dff/conf.py#L60-L107

• Is there any restrictions to fix broken/redirect links in old whatsnew/.rst?
• Is there any restrictions to fix broken/redirect links in old whatsnew/changelog.rst (i.e. Misc/NEWS)?

Not as such, although effort is usually prioritised for the newer/more active pages. Cleaning them up would mean linkcheck output is more useful in the future.

The way I see, this steps divide in:

• Clean BPO to GH redirection messages
• Fix broken links
• Clean CPython CVS URL redirections
• Clean GH issues to GH PR redirections
• Fix the rest of the occurrences.

...

• Should I create a single Pull Request for all the fixes?

Maybe a PR for each of these steps? And split smaller if you feel they're too big. Smaller PRs are easier to review and get merged.

[rffontenelle]

• 241 lines are redirection of CPython CVS URL, fixing /tree/ to /blob/ in GitHub URL

Please could you give some examples of these?

Sure. The file Doc/about.rst has :source:`Misc/ACKS` , which links to that file in CPython's source code repository. The custom role "source" is defined in Doc/tools/extensions/pyspecific.py to https://github.com/python/cpython/tree/main/ + the suffix.

GitHub URL for browsing files is /blob/ and directories is /tree/. When browsing Misc/ACKS, the /tree/ redirects to /blob/.

From linkcheck stdout:

about: line   33) redirect  https://github.com/python/cpython/tree/main/Misc/ACKS - permanently to https://github.com/python/cpython/blob/main/Misc/ACKS

See in the above log how the redirect was from /tree/ to /blob/.

• Is there any restrictions to fix broken/redirect links in old whatsnew/.rst?
• Is there any restrictions to fix broken/redirect links in old whatsnew/changelog.rst (i.e. Misc/NEWS)?

Not as such, although effort is usually prioritised for the newer/more active pages. Cleaning them up would mean linkcheck output is more useful in the future.

Great, I'll focus on the cleanup then.

• Should I create a single Pull Request for all the fixes?

Maybe a PR for each of these steps? And split smaller if you feel they're too big. Smaller PRs are easier to review and get merged.

Will do, thanks for the suggestion.

[hugovk]

Thanks, we can add the tree->blob redirects to linkcheck_allow_redirect like: https://github.com/python/devguide/blob/main/conf.py#L73-L74

[merwok]

Why not do the one-line fix to pyspecific rather?

[rffontenelle]

Because in :source:`something` the URL has /tree/ if something is a directory and /blob/ if it is a file. GitHub redirects both ways. If we put /blob/ in pyspecific.py, then a :source:`Misc/` would result in /blob/ being redirected to /tree/.

EDIT: See some examples:

$ curl -sI https://github.com/python/cpython/blob/main/Misc | grep -E '(^HTTP|^location:)'
HTTP/2 301 
location: https://github.com/python/cpython/tree/main/Misc
$ curl -sI https://github.com/python/cpython/tree/main/Misc/ACKS | grep -E '(^HTTP|^location:)'
HTTP/2 301 
location: https://github.com/python/cpython/blob/main/Misc/ACKS

[rffontenelle]

Suggestion please?

bpo-31453: Add TLSVersion constants and SSLContext.maximum_version / minimum_version attributes. The new API wraps OpenSSL 1.1 https://www.openssl.org/docs/man1.1.0/ssl/SSL_CTX_set_min_proto_version.html feature.

The text talks about OpenSSL 1.1.0 and the URL for version 1.1.0 is broken (HTTP 404), but there are valid URLs for the same resource pointing to OpenSSL version 1.1.1 and to 3.1.0. Should I update the URL to either one, or use wayback machine to make sure to match the OpenSSL version?

[merwok]

If openssl 1.1.1 is a bugfix release of the 1.1 branch, then it would be fine to point there.

[rffontenelle]

Created the first PR. I'm planning to create other 2 PRs, with more file changes than this one.

[rffontenelle]

I noticed the docs.yml workflow file runs for all branches from main to 3.7. Should I back-port the fixes to all these versions (cases were the bot won't be able to back-port automatically) or is there some code freeze that prevents it?

[merwok]

Backporting doc fixes to 3.11 is fine, but older branches only accept security fixes.

[rffontenelle]

Got it. It means a linkcheck job in docs.yml will require a condition check to exclude versions < 3.11. Thanks.

[merwok]

A workflow in the main branch applies to other branches?! What a strange system.

[rffontenelle]

Yep. It is a centralized way to control what routines will run in which events (push, pull requests, etc.) and in which branches.

[hugovk]

Are you sure?

I made a test repo with main and 3.11 branches:

• https://github.com/hugovk/test-workflows

Each branch has a workflow that does echo "hi from <branch-name>":

• https://github.com/hugovk/test-workflows/blob/main/.github/workflows/doc.yml
• https://github.com/hugovk/test-workflows/blob/3.11/.github/workflows/doc.yml

A push to each branch echoes the matching version number. So it runs its own workflow, not the one from main:

• https://github.com/hugovk/test-workflows/actions/runs/4720278815/jobs/8372127403
• https://github.com/hugovk/test-workflows/actions/runs/4720286253/jobs/8372144096

Also, making branches from each, and a PR back, also echoes the matching version number, not the one from main:

• Test PR into main hugovk/test-workflows#1
  • https://github.com/hugovk/test-workflows/actions/runs/4720297470/jobs/8372169004?pr=1
• Test PR into 3.11 hugovk/test-workflows#2
  • https://github.com/hugovk/test-workflows/actions/runs/4720299864/jobs/8372174031?pr=2

[rffontenelle]

I was sure until now. I mean, what's the point of having branch filters with they don't actually filter on which branch a workflow can or cannot be run?

on:
  push:
    branches:
      - main
      - '3.11'
      - '3.10'
    ...

[hugovk]

I expect it's to have as few changes as possible between branches to make it easier for the bot to backport (and for humans when the bot can't do it).

[rffontenelle]

Well, I stand corrected. That means a linkcheck job will need to be added to both main and 3.11. No problem.

[hugovk]

We probably don't want to run linkcheck on the CI for a couple of reasons:

• it takes about 50 minutes to run
• we can get new failures when things change external to a PR, like a website going down or temporary network failures

[rffontenelle]

Fixes to broken links now submitted.

[vstinner]

Fixed by ab80c6b. Backports will follow.

[rffontenelle]

@vstinner I was hoping to include at least another PR for "redirect with found" reported by linkcheck, (see above comment #103484 (comment)), and then we can close this. Can you please keep this open for now?