x build library/std compiler/rustc gives an error that proc_macro isn't built

[jyn514]

We started seeing a strange error to find the proc-macro crate during stage1 bootstrapping starting at this rust commit 0a7f2c3.
The command we're using is python3 x.py build library/std compiler/rustc src/tools/cargo.
The errors look like this:

...
Building stage1 compiler artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
...
error[E0463]: can't find crate for `proc_macro`
   --> /root/.cargo/registry/src/git.luolix.top-1ecc6299db9ec823/proc-macro-hack-0.5.19/src/lib.rs:150:1
    |
150 | extern crate proc_macro;
    | ^^^^^^^^^^^^^^^^^^^^^^^^ can't find crate

error[E0463]: can't find crate for `proc_macro`
   --> /root/.cargo/registry/src/git.luolix.top-1ecc6299db9ec823/proc-macro2-1.0.37/src/lib.rs:119:1
    |
119 | extern crate proc_macro;
    | ^^^^^^^^^^^^^^^^^^^^^^^^ can't find crate

   Compiling unic-emoji-char v0.9.0
error[E0463]: can't find crate for `proc_macro`

error[E0635]: unknown feature `proc_macro_span`
  --> /root/.cargo/registry/src/git.luolix.top-1ecc6299db9ec823/proc-macro2-1.0.37/src/lib.rs:90:59
   |
90 | #![cfg_attr(any(proc_macro_span, super_unstable), feature(proc_macro_span))]
   |                                                           ^^^^^^^^^^^^^^^

This is currently affecting our experimental rust + llvm@head build bot (https://buildkite.com/llvm-project/rust-llvm-integrate-prototype/builds/11778#0181d3d0-a73a-4852-8a31-edcc608fbb6f), but this error seems more general. I've set up a reproducer on Ubuntu in Docker over at https://github.com/krasimirgg/bootstrap-proc-macro-issue, which triggers this error via python3 x.py build library/std compiler/rustc src/tools/cargo using freshly imported rust.

It seems like python3 x.py build library/std compiler/rustc triggers this error; running python3 x.py build library/std or python3 x.py build compiler/rustc by themselves works.

Originally posted by @krasimirgg in #44293 (comment)

[jyn514]

#44293 (comment): build compiler/rustc implies build library/std; there should be no reason to use both in the same command. The error you're running into is since #95502 only std is being built, instead of all standard library crates (proc_macro doesn't depend on std).

That said, this is still a bug, compiler/rustc should be building the crates it cares about.

[jyn514]

Ok, I think I see the issue: StdLink is cached between the invocations, because we don't invalidate the cache when self.crates changes. Should be easy to fix, working on it now.