rustup 1.27 fails when installing targets on Windows in GitHub Actions

[VorpalBlade]

Problem

As of a couple of days ago it is no longer possible to install x86_64-pc-windows-gnu in github CI. This co-incides with the new rustup release 1.27.0 fairly closely.

I don't have Windows myself, so this is incredibly hard to debug for me (I can basically just tweak and see what happens in CI), but nothing has changed in the CI configuration on my side.

The error is:

Run rustup install --profile minimal stable && rustup default stable && rustup target add x86_64-pc-windows-gnu
info: syncing channel updates for 'stable-x86_64-pc-windows-msvc'

  stable-x86_64-pc-windows-msvc unchanged - rustc 1.76.0 (07dca489a 2024-02-04)

info: checking for self-update
info: downloading self-update
warning: tool `rust-analyzer` is already installed, remove it from `C:\Users\runneradmin\.cargo\bin`, then run `rustup update` to have rustup manage this tool.
warning: tool `rustfmt` is already installed, remove it from `C:\Users\runneradmin\.cargo\bin`, then run `rustup update` to have rustup manage this tool.
warning: tool `cargo-fmt` is already installed, remove it from `C:\Users\runneradmin\.cargo\bin`, then run `rustup update` to have rustup manage this tool.
error: invalid value 'C:\Users\runneradmin\.cargo\bin\rustup.exe' for '[+toolchain]': error: "C:\Users\runneradmin\.cargo\bin\rustup.exe" is not a valid subcommand, so it was interpreted as a toolchain name, but it is also invalid. To override the toolchain using the 'rustup +toolchain' syntax, make sure to prefix the toolchain override with a '+'

For more information, try '--help'.
Error: Process completed with exit code 1.

Here is an example where it fails: https://github.com/VorpalBlade/chezmoi_modify_manager/actions/runs/8255667914/job/22582675054

Linux and MacOS runners work just fine.

Steps

1. Run rustup install --profile minimal stable && rustup default stable && rustup target add x86_64-pc-windows-gnu in github CI on Windows

Possible Solution(s)

No response

Notes

No response

Rustup version

Run rustup --version
rustup 1.26.0 (5af9b9484 2023-04-05)
info: This is the version for the rustup toolchain manager, not the rustc compiler.

Installed toolchains

info: The currently active `rustc` version is `rustc 1.76.0 (07dca489a 2024-02-04)`
Default host: x86_64-pc-windows-msvc
rustup home:  C:\Users\runneradmin\.rustup
installed targets for active toolchain
--------------------------------------
i686-pc-windows-msvc
x86_64-pc-windows-gnu
x86_64-pc-windows-msvc
active toolchain
----------------
stable-x86_64-pc-windows-msvc (default)
rustc 1.76.0 (07dca489a 2024-02-04)

[VorpalBlade]

Adding --no-self-update to the install command seems to work around the issue. But I don't understand why only the -gnu target would be affected, not the -msvc target (they are both built in the same matrix).

[djc]

@ChrisDenton would you have any idea on what might be going on here?

[ChrisDenton]

Hm, I don't immediately see what the issue is. Both the working and failed commands look the same to me because it never gets as far as installing the target (as far as I can tell). Is there some other difference I'm missing?

I can try to repo it locally. Is there a link to download rustup 1.26?

[VorpalBlade]

Maybe my Windows-ignorance shows (I stopped using Windows after XP, all Linux since then), but I seem to remember that replacing or deleting currently open files on Windows is tricky (as in, you can't do it). That sounds like something a self-updater would run into. Or maybe I'm way off.

[ChrisDenton]

I believe rustup would warn about that and retry when that happens. Although maybe there are cases where it doesn't. The warnings it does give are suspicious though. It suggests that the binaries in cargo/bin aren't rustup proxies like they should be. Or if they are then rustup isn't recognizing them as such. Which might not be the issue but it is at least a point to start investigating.

The 1.27 release of rustup is very large with almost a years worth of commits so there have been a ton of changes which makes narrowing down what's responsible harder.

[VorpalBlade]

If you can't reproduce it locally, it is quite possibly github's default images has something weird in them. Likely you should be able to reproduce that in your own test repo as well (with shorter test cycles), but otherwise if you give me some test commands to put in (e.g. listing those suspicious directories) I'll give that a try (well tomorrow after work, I'm going to be immediately after writing this comment).

Fair warning, I don't remember much about windows, except you had to use dir instead of ls, so please give exact commands to use.

[ChrisDenton]

Oh I do get access denied errors: https://github.com/ChrisDenton/testme/actions/runs/8256694579/job/22585884277

info: downloading self-update
error: could not remove 'rustup-bin' file: 'C:\Users\runneradmin\.cargo\bin\rustup.exe'

Caused by:
    Access is denied. (os error 5)error: could not remove 'setup' file: 'C:\Users\runneradmin\.cargo\bin/rustup-init.exe'

Caused by:
    Access is denied. (os error 5)

It seems like maybe we're not retying on windows-gnu but we are on windows-msvc? That doesn't make sense to me though I'll take another look at the code. And I'm not sure why that would cause the warnings in the OP.

[ChrisDenton]

Oh, I'm not going crazy. It's not gnu that's the issue. It seems that the builders are racing with each other somehow. The two that go first win and the second two lose.

[ChrisDenton]

I just managed to reproduce the OP messages: https://github.com/ChrisDenton/testme/actions/runs/8256803265/job/22586213274

Seems random, which does suggest different builders can interfere with each other rather than having their own home directory. Rustup is not designed to be run concurrently (at the moment) but maybe rustup does need to take a lock when self updating.

[rami3l]

I guess running rustup set auto-self-update disable first might mitigate this issue, as per the user guide.

[ctron]

I am having similar issues with GitHub runners and rustup on Windows only. There seem to be different manifestations of this. And it all started when the new rustup version was released.

One error is:

Run rustup toolchain install stable --target wasm32-unknown-unknown
info: syncing channel updates for 'stable-x86_64-pc-windows-msvc'
info: latest update on 2024-02-08, rust version 1.76.0 (07dca489a 2024-02-04)
info: downloading component 'rust-std' for 'wasm32-unknown-unknown'
info: installing component 'rust-std' for 'wasm32-unknown-unknown'

  stable-x86_64-pc-windows-msvc updated - rustc 1.76.0 (07dca489a 2024-02-04) (from rustc 1.76.0 (07dca489a 2024-02-04))
info: checking for self-update

info: downloading self-update
warning: tool `rust-analyzer` is already installed, remove it from `C:\Users\runneradmin\.cargo\bin`, then run `rustup update` to have rustup manage this tool.
warning: tool `rustfmt` is already installed, remove it from `C:\Users\runneradmin\.cargo\bin`, then run `rustup update` to have rustup manage this tool.
warning: tool `cargo-fmt` is already installed, remove it from `C:\Users\runneradmin\.cargo\bin`, then run `rustup update` to have rustup manage this tool.
error: invalid value 'C:\Users\runneradmin\.cargo\bin\rustup.exe' for '[+toolchain]': error: "C:\Users\runneradmin\.cargo\bin\rustup.exe" is not a valid subcommand, so it was interpreted as a toolchain name, but it is also invalid. To override the toolchain using the 'rustup +toolchain' syntax, make sure to prefix the toolchain override with a '+'

Another one is:

Run rustup toolchain install stable --target wasm32-unknown-unknown
info: syncing channel updates for 'stable-x86_64-pc-windows-msvc'
info: latest update on 2024-02-08, rust version 1.76.0 (07dca489a 2024-02-04)
info: downloading component 'rust-std' for 'wasm32-unknown-unknown'
info: installing component 'rust-std' for 'wasm32-unknown-unknown'

  stable-x86_64-pc-windows-msvc updated - rustc 1.76.0 (07dca489a 2024-02-04) (from (timeout reading rustc version))

info: checking for self-update
info: downloading self-update
error: could not remove 'setup' file: 'C:\Users\runneradmin\.cargo\bin/rustup-init.exe': Access is denied. (os error 5)
warning: tool `rust-analyzer` is already installed, remove it from `C:\Users\runneradmin\.cargo\bin`, then run `rustup update` to have rustup manage this tool.
warning: tool `rustfmt` is already installed, remove it from `C:\Users\runneradmin\.cargo\bin`, then run `rustup update` to have rustup manage this tool.
warning: tool `cargo-fmt` is already installed, remove it from `C:\Users\runneradmin\.cargo\bin`, then run `rustup update` to have rustup manage this tool.
Error: Process completed with exit code 1.

Another one:

Run rustup toolchain install stable --target wasm32-unknown-unknown
info: syncing channel updates for 'stable-x86_64-pc-windows-msvc'
info: latest update on 2024-02-08, rust version 1.76.0 (07dca489a 2024-02-04)
info: downloading component 'rust-std' for 'wasm32-unknown-unknown'
info: installing component 'rust-std' for 'wasm32-unknown-unknown'

  stable-x86_64-pc-windows-msvc updated - rustc 1.76.0 (07dca489a 2024-02-04) (from rustc 1.76.0 (07dca489a 2024-02-04))

info: checking for self-update
info: downloading self-update
warning: tool `rust-analyzer` is already installed, remove it from `C:\Users\runneradmin\.cargo\bin`, then run `rustup update` to have rustup manage this tool.
warning: tool `rustfmt` is already installed, remove it from `C:\Users\runneradmin\.cargo\bin`, then run `rustup update` to have rustup manage this tool.
warning: tool `cargo-fmt` is already installed, remove it from `C:\Users\runneradmin\.cargo\bin`, then run `rustup update` to have rustup manage this tool.
error: could not remove 'setup' file: 'C:\Users\runneradmin\.cargo\bin/rustup-init.exe': Access is denied. (os error 5)
Error: Process completed with exit code 1.

To my understanding, github runners are isolated. And there is no other rustup running at that time.

[ChrisDenton]

I guess it might just be very hard to repo without CI.

It seems something in github is messing with the ~/,cargo/bin directory at the same time as rustup. If it's not a caching issue then maybe a scanner or some such.

It does seem to only happen in CI though.

[rami3l]

Looks like this probably has something to do with #2441?

[bematteo]

For the record, I have a single runner and it still happens like 50% of the times.
The runner is installed as a service using the NetworkService account.
(--no-self-update works as a workaround)

[rami3l]

The error messages comes from:

rustup/src/cli/self_update.rs

Lines 806 to 822 in 3f57299

	// If this file exists and is *not* equivalent to all other
	// preexisting tools we found, then we're going to assume that it
	// was preinstalled and actually pointing to a totally different
	// binary. This is intended for cases where historically users
	// ran `cargo install rustfmt` and so they had custom `rustfmt`
	// and `cargo-fmt` executables lying around, but we as rustup have
	// since started managing these tools.
	//
	// If the file is managed by rustup it should be equivalent to some
	// previous file, and if it's not equivalent to anything then it's
	// pretty likely that it needs to be dealt with manually.
	if tool_handles.iter().all(|h| *h != handle) {
	warn!("tool `{}` is already installed, remove it from `{}`, then run `rustup update` \
	to have rustup manage this tool.",
	tool, bin_path.display());
	continue;
	}

I have a feeling that this has something to do with how GitHub prepares its Windows images. For example, it might be tricking Rustup into thinking that the tools like rust-analyzer and rustfmt are installed by Rustup.

Anyway, without a proper reproduction on Windows, there's nothing we can do on our side.

[rbtcollins]

The warning about the proxies suggests a broken hardlink is occuring.

The race condition between concurrent installs is #988

[bematteo]

In my case I have a physical Windows machine with a single hosted runner, that runs one job at a time.
So I don't think it's necessarily the way GitHub prepares Windows images, or concurrent installs.

[ChrisDenton]

Can you give more details? Do you have a third party virus checker or other malware scanner? I've been trying to reproduce this locally but I've only managed in CI so far.

[rami3l]

As a mitigation has been implemented in #3907, starting from v1.28 this error will no longer happen (although updating from v1.27 to v1.28 doesn't count, since the business logic in question is executed before the update).

[djc]

@rami3l so probably okay to close this issue, then?

[rami3l]

@rami3l so probably okay to close this issue, then?

@djc Makes sense. Closing in favor of #988.