All x.py commands fail in bootstrap: cannot find file "ci-llvm/builder-config"

[RalfJung]

Trying to run ./x.py check miri fails right now:

$ ./x.py check miri
Building bootstrap
    Finished `dev` profile [unoptimized] target(s) in 0.05s
warning: unable to check if origin/master is old due to error: Not a directory (os error 20)
warning: origin/master is used to determine if files have been modified
warning: if it is not updated, this may cause files to be needlessly reformatted
thread 'main' panicked at src/core/config/config.rs:1237:13:
fs::read_to_string(file) failed with No such file or directory (os error 2) ("config file /home/r/src/rust/rustc.2/build/x86_64-unknown-linux-gnu/ci-llvm/builder-config not found")
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Build completed unsuccessfully in 0:00:00

It works fine in other checkouts though. The error is correct, there is no such file -- so I guess some other part of bootstrap failed to create that file?

x.py test ui fails with the same error.

I am on commit 263a3ae, git status is clean.
Reverting back to 085744b fixes this, so seems like #129788 is the cause of the problem.

Cc @rust-lang/bootstrap

[Kobzol]

I think that this is the issue:

warning: unable to check if origin/master is old due to error: Not a directory (os error 20)
warning: origin/master is used to determine if files have been modified
warning: if it is not updated, this may cause files to be needlessly reformatted

as bootstrap probably uses an old ci-llvm version in your checkout. Can you configure the remote? Otherwise you might be using an old/weird version of ci-llvm anyway, regardless of the new error.

[RalfJung]

The remote is configured. The warning just can't deal with worktrees (#130111). But I think that's orthogonal as I got the warning already before that most recent PR. I don't think #130111 also affects fetching the right LLVM version.

[Kobzol]

Ah, I see, it's the worktree issue. Well, it is orthogonal in a way, but the issue is that bootstrap now assumes that you have a recent-ish ci-llvm archive, because we have added some config file to it that is now being checked by bootstrap. Since in your case, bootstrap can't detect the master commit, is is probably also unable to download the correct (recent) ci-llvm archive. Therefore, it probably uses some old archive that misses the config file, and that causes bootstrap to fail.

When implementing the config checking, we assumed that if you are on a commit that includes the new checks, then you also have to use a recent ci-llvm. But if git detection is somehow not working, then it's possible to be on a new commit that performs the check, but also to have an old ci-llvm archive at the same time.

[RalfJung]

Bootstrap can detect the master commit just fine I think. ci-llvm has worked perfectly fine in worktrees for years. The warning has its own logic for checking whether the master commit is outdated, and only that logic is faulty.

[Kobzol]

Sorry, then I got it wrong, it was a red herring. I actually just checked that it also doesn't work for me (nor anyone else, probably). Let's quickly revert #129788 then.

[RalfJung]

That said, something also changed with remote branch logic recently that broke it in worktrees -- ./x.py fmt did the right thing on worktrees for years, but now it formats >5k files.

[onur-ozkan]

How exactly can we reproduce this? Running x check miri on a fresh build works for me.

[onur-ozkan]

I assume you have the old ci-llvm directory where it was unpacked with old bootstrap logic.

[Kobzol]

I see. It works when you newly extract ci-llvm, but not with an already extracted directory. So removing the build directory (or just doing x clean) is a workaround.

[RalfJung]

rm build/host/ci-llvm/ -rf suffices and avoids having to re-download and re-build everything.