Support for "cargo vendor" broke -Zbuild-std for custom targets. Or at least, the current behavior is now different

[ivmarkov]

(Disclaimer: Rust noob here. Don't kill me if I've misinterpreted a correct behaviour.)

Problem
Building a crate with -Zbuild-std=... and --target <target> where <target> is a custom target linked in Rustup like this (rustup link mytarget /...path.../) is broken since approx the time when the changes for "cargo vendor" landed.

Steps

1. Build a custom rust toolchain. For example, this one.
2. Register the just-built toolchain as a custom target in rustup as described in the above repo.
3. Try to build a crate (for example, this one) by using cargo build -Z build-std=std,panic_abort -Z build-std-features=panic_immediate_abort
4. Problem 1: Building the crate will fail with complaint error: could not read vendor path. Apparently, because of this code on line 55.
5. Problem 2: If you make an empty "vendor" dir and re-run the build, it will fail again now with the dreaded:

error[E0463]: can't find crate for `std`
  |
  = note: the `xtensa-esp32-none-elf` target may not be installed

... which is due to the fact that cargo does not pick up the std lib from the sysroot of the custom target at all. An indication of that is the fact that it downloads and compiles the rustc-std-workspace-* crates from GitHub rather than picking up their patches from the sysroot of the custom target. I.e. when run with --verbose:

rustc --crate-name rustc_std_workspace_alloc --edition=2018 /Users/...path.../.cargo/registry/src/git.luolix.top-1ecc6299db9ec823/rustc-std-workspace-alloc-1.0.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no -C metadata=c8dee4e2d9f82246 -C extra-filename=-c8dee4e2d9f82246 --out-dir /Users/<censored>/rust-esp32-std-hello/rust/target/xtensa-esp32-none-elf/release/deps --target xtensa-esp32-none-elf -Z force-unstable-if-unmarked -L dependency=/Users/...path.../rust-esp32-std-hello/rust/target/xtensa-esp32-none-elf/release/deps -L dependency=/Users/...path.../rust-esp32-std-hello/rust/target/release/deps --cap-lints allow (exit code: 1)

Apparently, the above is caused by the logic here (line 67 and onward), but I've not analyzed (yet) what needs to be fixed in detail.

Possible Solution(s)
The following workaround seems to fix the issue:

1. cd rust
2. mkdir vendor
3. cd vendor
4. ln -s ../library/rustc-std-workspace-alloc rustc-std-workspace-alloc
5. ln -s ../library/rustc-std-workspace-core rustc-std-workspace-core
6. ln -s ../library/rustc-std-workspace-std rustc-std-workspace-std

Yet, this just does not feel right.

I just want to use my custom target with -Zbuild-std and that's it. Ideally, I should not be even aware of the vendor machinery, let alone symlinking the rustc-std-workspace-* patch crates in there.

Or is this how it all is supposed to work with custom targets, now that cargo vendor has landed?

[alexcrichton]

The changes to cargo vendor is that we expect the Rust libstd sysroot crates to be vendored in the source directory. This probably means that we either need specific code for "you're working with a local toolchain" or the Rust build system needs a new build target for "make sure to build a source tree that's compatible with cargo -Zbuild-std"

[ivmarkov]

The changes to cargo vendor is that we expect the Rust libstd sysroot crates to be vendored in the source directory.

What does that mean? When working with custom targets, cargo is already very well aware of the following two locations:

• <rust-github-copy>/library
• <rust-github-copy>/vendor (this one actually does not really exist with custom targets, but cargo tries to access it anyway)

It is aware of those because when the custom target is build (e.g. ./x.py build --stage2), there is a symlink autocreated as follows:

<rust-github-copy>/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/src/rust -> symlinks back to <rustc-github-copy>/.

... where the custom target root is <rust-github-copy>/build/x86_64-unknown-linux-gnu/stage2 (as I'm not installing the custom target after building it but just linking it into Rustup by doing rustup link xtensa <rust-github-copy>/build/x86_64-unknown-linux-gnu/stage2)

So to summarize Cargo does have access to the /library and /vendor dirs even with custom targets. It also knows that the 'std' and 'core' code is in /library and not in /vendor. However - since the changes that introduced cargo vendor - it somehow does expect the rustc-std-workspace-* hacks specifically to be in /vendor while they are are actually inside /library. (Hence why I have to symlink them from inside /vendor in my workaround.)

Why are the rustc-std-workspace-* crates now expected to be in /vendor, and only these are expected to be in /vendor from all the std/crate/etc. machinery while the rest is still (correctly) looked up from /library?

Sounds like a bug to me?

[alexcrichton]

Yes I believe your diagnosis of the situation is all correct, and this is due to changes in #8834. Cargo doesn't expect the rustc-std-workspace crates to be in vendor themselves, but it's expecting the copies from crates.io is vendored in there, since that's what cargo vendor generates.

It's not clear if this should be a bug for Cargo to fix or a bug for rust-lang/rust and/or rustup to have an extra step when you create a local toolchain.

[ehuss]

I'm going to close this as it has been reverted in #8968. Will continue tracking in rust-lang/wg-cargo-std-aware#23.