CI: Bootstrap no longer documents compiler crates

[fmease]

Bootstrap no longer documents any rustc_* / compiler/ crates or rustdoc in CI.
The index now only contains a meager 20 or so crates.
This has to have regressed fairly recently.

[Nemo157]

It regressed in the latest nightly, we have CI tests on docs.rs checking for some of these and the last daily test succeeded.

[ChrisDenton]

If I'm reading the commit history right there were a number of PRs that touched one of CI, bootstrap or rustdoc itself:

• rustdoc: elide cross-crate default generic arguments #112463
• bootstrap: bump fd-lock, clap and windows #116939
• update which targets we test Miri on #117267
• pass CODEGEN_BACKENDS to docker #117328
• More rustc_interface cleanups #117376
• improve and fix x install #117383

[Mark-Simulacrum]

This appears to have regressed in #117328... but I'm pretty confused by that.

$ for commit in $(git log --first-parent --format=%H | head -n30); do echo $commit; curl -sI "https://ci-artifacts.rust-lang.org/rustc-builds/$commit/rustc-docs-nightly-x86_64-unknown-linux-gnu.tar.xz" | grep content-length; done
ffb7ed9fa420e9bcd98d84b431b2009445b7b967
content-length: 5454792
650991d62c3a2c80ba27009d06839adbb038bf5e
content-length: 5456880
a395214a3a88be976eec9256987d5eb6b3584bdc
content-length: 5454420
31bc7e2c47e82798a392c770611975a6883132c8
content-length: 5456600
236ac911de2f4b6210ba99e25c6c4843c9381a1a
content-length: 5454984
e6e931dda5fffbae0fd87c5b1af753cc95556880
content-length: 5468104
6d1fc53cf4f61b0dce5fc84cebae3b9fb8a45436
content-length: 5468832
e324cf0f73f09388d494986550f46dc9e12badd3
content-length: 5468796
91bbdd927a5e53a2fe126304fe8adbedf339616c
content-length: 5468436
1dfb6b162be402d8ca37e8aad4f58898b44e3a15
content-length: 5468796
bcb5798dd890a691644af9d371f3bd7fcc465584
content-length: 22395756
608e9682f0a6482903de8d3332770104a0ad943c
content-length: 22390916
ec2b311914a4880235862f3cd65fe24e92e6d3a5
content-length: 22391560
88ae8c93854a1a67c6c0a45971c7198477456c33
content-length: 22396416
83c9732e0c7e1cae5f039677da4c51ee1d9b19b0
content-length: 22393792
160fd3ceb5143100d4e06f324b02af1f9e786372
content-length: 22375124
bbcc1691a4646a6d2ebb2189bdff5b878407fbe6

[lqd]

This appears to have regressed in #117328... but I'm pretty confused by that.

Since #117328 mostly allowed #81746 to have an effect, it could be that the issue is in #81746.

[Noratrieb]

cc @bjorn3 and also @onur-ozkan maybe you happen to have an idea too

[bjorn3]

I don't have a clue how either PR could have affected building docs.

[onur-ozkan]

I would do quick check between the recently merged PRs as there are not many. It seems not to be so hard to find and fix the problem. But I can only do that later tonight.. I don't have access to my computers right now.

[onur-ozkan]

Ok, so the problem is when we use cranelift as a backend with llvm, documentation outputs generated by these backends seem to conflict with each other during the invocations. It appears that one of them(most likely cranelift) is not generating the compiler documentation. As we purge the previously generated documentation output at

rust/src/bootstrap/src/core/builder.rs

Line 1290 in 50be229

self.clear_if_dirty(&my_out, &rustdoc);

we end up with output that doesn't have the compiler documentation.

I am currently unsure of the exact reason why cranelift is causing this issue. I will open a temporary fix PR and investigate further once I get more time.

[RalfJung]

Indeed the search at https://doc.rust-lang.org/nightly/nightly-rustc/ is completely useless right now. If a fix would take more than a day, reverting the offending PR should also be considered, given that those docs are quite crucial for rustc contributors.

[onur-ozkan]

Indeed the search at https://doc.rust-lang.org/nightly/nightly-rustc/ is completely useless right now. If a fix would take more than a day, reverting the offending PR should also be considered, given that those docs are quite crucial for rustc contributors.

Fix PR is already open at #117471

[RalfJung]

Yeah I've seen that, but I can't tell if that's the lowest-risk approach. 🤷 Reverting (or un-setting CODEGEN_BACKENDS on whatever builder builds the docs) is something we know will obviously work.

[onur-ozkan]

I am quite sure that PR contains no risk, but I am fine with reverting codegen environment variable updates in the runner containers

[onur-ozkan]

#117471 did fix it locally, but seems didn't fix it on actual environment..

[onur-ozkan]

To fix this, we can either disable cranelift on the gh action step which we build nightly documentation or revert #117328 until we find out and fix the problem of compiler documentations with cranelift.

[RalfJung]

disable cranelift on the gh action step which we build nightly documentation

Which one is that?

[onur-ozkan]

disable cranelift on the gh action step which we build nightly documentation

Which one is that?

I guess (never worked on the infra side before) these ones

rust/.github/workflows/ci.yml

Lines 537 to 542 in 2429818

	- name: upload artifacts to S3
	run: src/ci/scripts/upload-artifacts.sh
	env:
	AWS_ACCESS_KEY_ID: "${{ env.ARTIFACTS_AWS_ACCESS_KEY_ID }}"
	AWS_SECRET_ACCESS_KEY: "${{ secrets[format('AWS_SECRET_ACCESS_KEY_{0}', env.ARTIFACTS_AWS_ACCESS_KEY_ID)] }}"
	if: "success() && !env.SKIP_JOB && (github.event_name == 'push' || env.DEPLOY == '1' || env.DEPLOY_ALT == '1')"

rust/.github/workflows/ci.yml

Lines 661 to 666 in 2429818

	- name: upload artifacts to S3
	run: src/ci/scripts/upload-artifacts.sh
	env:
	AWS_ACCESS_KEY_ID: "${{ env.ARTIFACTS_AWS_ACCESS_KEY_ID }}"
	AWS_SECRET_ACCESS_KEY: "${{ secrets[format('AWS_SECRET_ACCESS_KEY_{0}', env.ARTIFACTS_AWS_ACCESS_KEY_ID)] }}"
	if: "success() && !env.SKIP_JOB && (github.event_name == 'push' || env.DEPLOY == '1' || env.DEPLOY_ALT == '1')"

rust/.github/workflows/ci.yml

Lines 160 to 165 in 2429818

	- name: upload artifacts to S3
	run: src/ci/scripts/upload-artifacts.sh
	env:
	AWS_ACCESS_KEY_ID: "${{ env.ARTIFACTS_AWS_ACCESS_KEY_ID }}"
	AWS_SECRET_ACCESS_KEY: "${{ secrets[format('AWS_SECRET_ACCESS_KEY_{0}', env.ARTIFACTS_AWS_ACCESS_KEY_ID)] }}"
	if: "success() && !env.SKIP_JOB && (github.event_name == 'push' || env.DEPLOY == '1' || env.DEPLOY_ALT == '1')"

[RalfJung]

Those steps just do the uploading, the docs are most likely created before. We need to figure out which runner is doing this and disable cranelift there.

[onur-ozkan]

It's done in multiple place(creating and uploading). I think the question is which one is getting deployed to https://doc.rust-lang.org/nightly/nightly-rustc/index.html.

[RalfJung]

It's been 3 days without docs, clearly this is not entirely trivial to fix. Let's revert.
#117535

[lqd]

The docs are back.

[ChrisDenton]

While the immediate issue is addressed, do we need an issue to track the underlying problem?

[lqd]

We could. At the same time, @bjorn3 and @onur-ozkan are also still working on this, e.g. in this zulip thread as it's quite important to fix. There's little chance we'll forget about this, and it should hopefully be fixed very soon.

[onur-ozkan]

The underlying problem is that symlinks are being dropped in a wrong way between invocations. I can easily fix this by copying the document outputs instead of linking them, but I would prefer to invest a couple more hours in fixing the linking issue rather than replacing it.