Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch to using the v2 resolver in the workspace #128359

Closed
wants to merge 5 commits into from

Conversation

tgross35
Copy link
Contributor

@tgross35 tgross35 commented Jul 29, 2024

Pinning the resolver to v1 was done in 5abff37 ("Explicit set workspace.resolver ...") in order to suppress warnings. Since there is no specific reason not to use the new resolver and since it fixes issues, change to resolver = "2".

Fixes #128358

r? @ehuss

@rustbot rustbot added A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue. labels Jul 29, 2024
@rustbot
Copy link
Collaborator

rustbot commented Jul 29, 2024

Portable SIMD is developed in its own repository. If possible, consider making this change to rust-lang/portable-simd instead.

cc @calebzulawski, @programmerjake

@tgross35 tgross35 force-pushed the use-new-resolver branch 2 times, most recently from 4bfb06f to 43a47fb Compare July 29, 2024 19:00
@tgross35 tgross35 changed the title Update the workspace to use the new resolver Switch to using the v2 resolver in the workspace Jul 29, 2024
@rust-log-analyzer

This comment has been minimized.

@rust-log-analyzer

This comment has been minimized.

@rustbot
Copy link
Collaborator

rustbot commented Jul 29, 2024

These commits modify the Cargo.lock file. Unintentional changes to Cargo.lock can be introduced when switching branches and rebasing PRs.

If this was unintentional then you should revert the changes before this PR is merged.
Otherwise, you can ignore this comment.

@rust-log-analyzer

This comment has been minimized.

@tgross35
Copy link
Contributor Author

@TimNN in #128358 you mentioned you were able to try with the v2 resolver. Did you have to do anything special? It seems like this gets stuck on multiple versions of std.

@TimNN
Copy link
Contributor

TimNN commented Jul 30, 2024

building or checking individual targets seems to work fine for me. However, I can reliably reproduce the CI failure by doing a plain x check without specifying any targets.

I haven't figured out yet which targets are apparently incompatible (or if there's a specific problem with x check without any arguments.

@TimNN
Copy link
Contributor

TimNN commented Jul 30, 2024

I found out what makes x check fail due to two different std versions:

  • For a plain x check invocation, std gets built twice:
    • Once as part of checking std
    • Once as part of checking sysroot
  • For each of these builds, bootstrap copies libstd into the sysroot directory
  • The sysroot enables std/panic_unwind by default; plain std does not
    • Thus two different versions of std

If I hack around this by forcing plain std to be built with panic-unwind (i.e., add cargo.arg("--features=std/panic-unwind"); after this), then I get a successful x check run with resolver = "2".

The solutions I can think of are:

  1. Ensure that plain std and sysroot check std with the same features.
  2. Figure out why plain x check checks std and sysroot separately, whereas x check <targets> seems to check them together, and fix that.
  3. Make bootstrap only copy libstd to the sysroot from either checking std or sysroot, but not from both.

edit: For the record, and in case anyone is interested, how I got here:
  • Make sure build/<target>/stage0-std/<target>/release/deps/ is empty, so cargo rebuilds
  • MAGIC_EXTRA_RUSTFLAGS=-Ztreat-err-as-bug x -j1 check -v
    • -Ztreat-err-as-bug because rustc produces thousands of errors if it cannot load std
    • -j1 also ensures there's exactly one things that fails, not multiple concurrent failures.
  • rustc will output the conflicting "candidate" rmeta files, including their hashes.
  • Search the -v output for the rustc invocations corresponding to the two hashes.
  • Diff the two invocations.
    • The main difference I found was --cfg 'feature="panic-unwind"' being present in only one invocation.
    • -C prefer-dynamic also differed between the two invocations, but I just hoped that that wasn't relevant (which it wasn't).

@tgross35
Copy link
Contributor Author

@rust-log-analyzer

This comment has been minimized.

@bors
Copy link
Contributor

bors commented Aug 2, 2024

☔ The latest upstream changes (presumably #128529) made this pull request unmergeable. Please resolve the merge conflicts.

@@ -67,6 +67,7 @@ impl Step for Std {
target,
self.override_build_kind.unwrap_or(builder.kind),
);
cargo.arg("--features=std/panic-unwind");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the root cause why this is necessary is that the sysroot crate enables the panic_unwind feature of std, rather than the panic-unwind feature. The panic_unwind feature exists because the panic-unwind feature enables the panic_unwind dependency using panic-unwind = ["panic_unwind"] rather than panic-unwind = ["dep:panic_unwind"].

Copy link
Member

@bjorn3 bjorn3 Aug 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Try this diff:

diff --git a/library/std/Cargo.toml b/library/std/Cargo.toml
index fe601855cc1..a025a4ea04e 100644
--- a/library/std/Cargo.toml
+++ b/library/std/Cargo.toml
@@ -90,8 +90,8 @@ backtrace = [
     'miniz_oxide/rustc-dep-of-std',
 ]
 
-panic-unwind = ["panic_unwind"]
-profiler = ["profiler_builtins"]
+panic-unwind = ["dep:panic_unwind"]
+profiler = ["dep:profiler_builtins"]
 compiler-builtins-c = ["alloc/compiler-builtins-c"]
 compiler-builtins-mem = ["alloc/compiler-builtins-mem"]
 compiler-builtins-no-asm = ["alloc/compiler-builtins-no-asm"]
diff --git a/library/sysroot/Cargo.toml b/library/sysroot/Cargo.toml
index 7165c3e48af..b05f61fedcd 100644
--- a/library/sysroot/Cargo.toml
+++ b/library/sysroot/Cargo.toml
@@ -20,7 +20,7 @@ compiler-builtins-no-f16-f128 = ["std/compiler-builtins-no-f16-f128"]
 compiler-builtins-mangled-names = ["std/compiler-builtins-mangled-names"]
 llvm-libunwind = ["std/llvm-libunwind"]
 system-llvm-libunwind = ["std/system-llvm-libunwind"]
-panic-unwind = ["std/panic_unwind"]
+panic-unwind = ["std/panic-unwind"]
 panic_immediate_abort = ["std/panic_immediate_abort"]
 optimize_for_size = ["std/optimize_for_size"]
 profiler = ["std/profiler"]

(the changes to the profiler feature are not strictly necessary, but I figured removing another unintentional implicit feature can't hurt)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, added at 7446e7f

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It may work without the cargo.arg("--features=std/panic-unwind"); now.

Pinning the resolver to v1 was done in 5abff37 ("Explicit set
workspace.resolver ...") in order to suppress warnings. Since there is
no specific reason not to use the new resolver and since it fixes
issues, change to `resolver = "2"`.

Fixes: rust-lang#128358
With the v2 resolver, this has the intended behavior of building
compiler_builtins. Unfortunately, this exposes the fact that the current
stage0 compiler cannot build the `f16` and `f128` implementation of
compiler_builtins.

Unconditionally disable `f16` and `f128` for now so we can update the
resolver.
`std/panic-unwind` gets enabled when building the sysroot, but not when
checking `std`. This was not a problem with the v1 resolver because it
would unify features and just always enable `panic-unwind`.

With the v2 resolver, however, this causes `std` to get built twice with
different sets of features. This then causes an "multiple candidates for
`rmeta` dependency `std` found" error.

Always enable the `panic-unwind` feature to avoid this conflict.
From: bjorn3 <17426603+bjorn3@users.noreply.github.com>
@rust-log-analyzer
Copy link
Collaborator

The job mingw-check-tidy failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

COPY host-x86_64/mingw-check/validate-toolstate.sh /scripts/
COPY host-x86_64/mingw-check/validate-error-codes.sh /scripts/

# NOTE: intentionally uses python2 for x.py so we can test it still works.
# validate-toolstate only runs in our CI, so it's ok for it to only support python3.
ENV SCRIPT TIDY_PRINT_DIFF=1 python2.7 ../x.py test \
           --stage 0 src/tools/tidy tidyselftest --extra-checks=py:lint,cpp:fmt
# This file is autogenerated by pip-compile with Python 3.10
# by the following command:
#
#    pip-compile --allow-unsafe --generate-hashes reuse-requirements.in
---
#13 3.035 Building wheels for collected packages: reuse
#13 3.036   Building wheel for reuse (pyproject.toml): started
#13 3.298   Building wheel for reuse (pyproject.toml): finished with status 'done'
#13 3.299   Created wheel for reuse: filename=reuse-4.0.3-cp310-cp310-manylinux_2_35_x86_64.whl size=132715 sha256=dfa09868353292d98f811d3efdb0d54d07389e808efc71d68e3b93c514bf8bec
#13 3.300   Stored in directory: /tmp/pip-ephem-wheel-cache-f9ojf3sd/wheels/3d/8d/0a/e0fc6aba4494b28a967ab5eaf951c121d9c677958714e34532
#13 3.303 Installing collected packages: boolean-py, binaryornot, tomlkit, reuse, python-debian, markupsafe, license-expression, jinja2, chardet, attrs
#13 3.731 Successfully installed attrs-23.2.0 binaryornot-0.4.4 boolean-py-4.0 chardet-5.2.0 jinja2-3.1.4 license-expression-30.3.0 markupsafe-2.1.5 python-debian-0.1.49 reuse-4.0.3 tomlkit-0.13.0
#13 3.733 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
#13 4.304 Collecting virtualenv
#13 4.304 Collecting virtualenv
#13 4.467   Downloading virtualenv-20.26.3-py3-none-any.whl (5.7 MB)
#13 4.680      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.7/5.7 MB 27.0 MB/s eta 0:00:00
#13 4.742 Collecting filelock<4,>=3.12.2
#13 4.749   Downloading filelock-3.15.4-py3-none-any.whl (16 kB)
#13 4.771 Collecting distlib<1,>=0.3.7
#13 4.779   Downloading distlib-0.3.8-py2.py3-none-any.whl (468 kB)
#13 4.796      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 468.9/468.9 KB 32.5 MB/s eta 0:00:00
#13 4.830 Collecting platformdirs<5,>=3.9.1
#13 4.838   Downloading platformdirs-4.2.2-py3-none-any.whl (18 kB)
#13 4.926 Installing collected packages: distlib, platformdirs, filelock, virtualenv
#13 5.119 Successfully installed distlib-0.3.8 filelock-3.15.4 platformdirs-4.2.2 virtualenv-20.26.3
#13 DONE 5.2s

#14 [7/8] COPY host-x86_64/mingw-check/validate-toolstate.sh /scripts/
#14 DONE 0.0s
---
DirectMap4k:      227264 kB
DirectMap2M:     9209856 kB
DirectMap1G:     9437184 kB
##[endgroup]
Executing TIDY_PRINT_DIFF=1 python2.7 ../x.py test            --stage 0 src/tools/tidy tidyselftest --extra-checks=py:lint,cpp:fmt
+ TIDY_PRINT_DIFF=1 python2.7 ../x.py test --stage 0 src/tools/tidy tidyselftest --extra-checks=py:lint,cpp:fmt
    Finished `dev` profile [unoptimized] target(s) in 0.04s
##[endgroup]
downloading https://ci-artifacts.rust-lang.org/rustc-builds-alt/53676730146e38e4697b6204c2ee61a9fd6b7e51/rust-dev-nightly-x86_64-unknown-linux-gnu.tar.xz
extracting /checkout/obj/build/cache/llvm-53676730146e38e4697b6204c2ee61a9fd6b7e51-true/rust-dev-nightly-x86_64-unknown-linux-gnu.tar.xz to /checkout/obj/build/x86_64-unknown-linux-gnu/ci-llvm
---
##[endgroup]
fmt check
fmt: checked 5418 files
tidy check
dependency exception `wasm-encoder` license has changed
    previously `Apache-2.0 WITH LLVM-exception` now `Apache-2.0 WITH LLVM-exception OR Apache-2.0 OR MIT`
dependency exception `wasm-metadata` license has changed
dependency exception `wasm-metadata` license has changed
    previously `Apache-2.0 WITH LLVM-exception` now `Apache-2.0 WITH LLVM-exception OR Apache-2.0 OR MIT`
dependency exception `wasmparser` license has changed
dependency exception `wasmparser` license has changed
    previously `Apache-2.0 WITH LLVM-exception` now `Apache-2.0 WITH LLVM-exception OR Apache-2.0 OR MIT`
    update EXCEPTIONS for the new license
dependency exception `wast` license has changed
    previously `Apache-2.0 WITH LLVM-exception` now `Apache-2.0 WITH LLVM-exception OR Apache-2.0 OR MIT`
    update EXCEPTIONS for the new license
dependency exception `wat` license has changed
    previously `Apache-2.0 WITH LLVM-exception` now `Apache-2.0 WITH LLVM-exception OR Apache-2.0 OR MIT`
dependency exception `wit-component` license has changed
dependency exception `wit-component` license has changed
    previously `Apache-2.0 WITH LLVM-exception` now `Apache-2.0 WITH LLVM-exception OR Apache-2.0 OR MIT`
dependency exception `wit-parser` license has changed
dependency exception `wit-parser` license has changed
    previously `Apache-2.0 WITH LLVM-exception` now `Apache-2.0 WITH LLVM-exception OR Apache-2.0 OR MIT`
    update EXCEPTIONS for the new license
tidy error: invalid license `Apache-2.0 WITH LLVM-exception` in `registry+https://github.com/rust-lang/crates.io-index#wasi-preview1-component-adapter-provider@23.0.1`
tidy error: invalid license `BSD-2-Clause` in `registry+https://github.com/rust-lang/crates.io-index#zerocopy@0.6.6`
tidy error: invalid license `BSD-2-Clause` in `registry+https://github.com/rust-lang/crates.io-index#zerocopy-derive@0.6.6`
tidy error: invalid license `BSD-2-Clause` in `registry+https://github.com/rust-lang/crates.io-index#zerocopy@0.6.6`
tidy error: invalid license `BSD-2-Clause` in `registry+https://github.com/rust-lang/crates.io-index#zerocopy-derive@0.6.6`
dependency exception `wasm-encoder` license has changed
    previously `Apache-2.0 WITH LLVM-exception` now `Apache-2.0 WITH LLVM-exception OR Apache-2.0 OR MIT`
dependency exception `wasm-metadata` license has changed
dependency exception `wasm-metadata` license has changed
    previously `Apache-2.0 WITH LLVM-exception` now `Apache-2.0 WITH LLVM-exception OR Apache-2.0 OR MIT`
dependency exception `wasmparser` license has changed
dependency exception `wasmparser` license has changed
    previously `Apache-2.0 WITH LLVM-exception` now `Apache-2.0 WITH LLVM-exception OR Apache-2.0 OR MIT`
    update EXCEPTIONS for the new license
dependency exception `wast` license has changed
    previously `Apache-2.0 WITH LLVM-exception` now `Apache-2.0 WITH LLVM-exception OR Apache-2.0 OR MIT`
    update EXCEPTIONS for the new license
dependency exception `wat` license has changed
    previously `Apache-2.0 WITH LLVM-exception` now `Apache-2.0 WITH LLVM-exception OR Apache-2.0 OR MIT`
dependency exception `wit-component` license has changed
dependency exception `wit-component` license has changed
    previously `Apache-2.0 WITH LLVM-exception` now `Apache-2.0 WITH LLVM-exception OR Apache-2.0 OR MIT`
dependency exception `wit-parser` license has changed
dependency exception `wit-parser` license has changed
    previously `Apache-2.0 WITH LLVM-exception` now `Apache-2.0 WITH LLVM-exception OR Apache-2.0 OR MIT`
    update EXCEPTIONS for the new license
tidy error: invalid license `Apache-2.0 WITH LLVM-exception` in `registry+https://github.com/rust-lang/crates.io-index#wasi-preview1-component-adapter-provider@23.0.1`
tidy error: invalid license `BSD-2-Clause` in `registry+https://github.com/rust-lang/crates.io-index#zerocopy@0.6.6`
tidy error: invalid license `BSD-2-Clause` in `registry+https://github.com/rust-lang/crates.io-index#zerocopy-derive@0.6.6`
removing old virtual environment
removing old virtual environment
creating virtual environment at '/checkout/obj/build/venv' using 'python3.10'
Requirement already satisfied: pip in ./build/venv/lib/python3.10/site-packages (24.1)
  Downloading pip-24.2-py3-none-any.whl.metadata (3.6 kB)
Downloading pip-24.2-py3-none-any.whl (1.8 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 16.7 MB/s eta 0:00:00
Installing collected packages: pip
---
All checks passed!
checking C++ file formatting
some tidy checks failed

Command LD_LIBRARY_PATH="/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-bootstrap-tools/x86_64-unknown-linux-gnu/release/deps:/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/lib" RUSTC="/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/rustc" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/rust-tidy" "/checkout" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "/checkout/obj/build" "4" "--extra-checks=py:lint,cpp:fmt" (failure_mode=DelayFail) did not execute successfully.
Created at: src/core/build_steps/tool.rs:1113:23
Executed at: src/core/build_steps/test.rs:1078:29

Build completed unsuccessfully in 0:01:20

@tgross35
Copy link
Contributor Author

tgross35 commented Aug 2, 2024

The tidy error looks like zerocopy 0.6.6 is only BSD-2, 0.7.34 is BSD-2 or Apache-2 or MIT. This dependency seems to come from ppv-lite86. Seems like the previous resolver just used 0.7 for everything?

I think we will need a release after cryptocorrosion/cryptocorrosion#77 to resolve this, or temporarily allow BSD-2.

@tgross35
Copy link
Contributor Author

tgross35 commented Aug 2, 2024

Still the duplicate lang item failure on panic_impl https://github.com/rust-lang/rust/actions/runs/10218524144/job/28274840523?pr=128359#step:25:4210

@bjorn3
Copy link
Member

bjorn3 commented Aug 2, 2024

I believe the duplicate lang item error is caused by the presence of cargo.arg("--features=std/panic-unwind");. While working on #128534 I accidentally reproduced that exact same issue by cherry-picking your commit to add it in the hope that it would fix another problem. (The problem I tried to fix turned out to be caused by me messing up the exclude key in the new workspace I introduced)

@tgross35
Copy link
Contributor Author

tgross35 commented Aug 2, 2024

Does changing to the v2 resolver "just work" with the separate workspaces? I think that sounds like a nicer solution.

@bjorn3
Copy link
Member

bjorn3 commented Aug 2, 2024

The patch from #128359 (comment) is still necessary with #128534. And something like ./x.py check alloc is broken too when I switch to the v2 resolver:

$ ./x.py check alloc
Building bootstrap
    Finished `dev` profile [unoptimized] target(s) in 0.05s
Checking stage0 library artifacts {alloc} (aarch64-unknown-linux-gnu)
error: none of the selected packages contains these features: backtrace, panic-unwind
Build completed unsuccessfully in 0:00:00

@bors
Copy link
Contributor

bors commented Aug 4, 2024

☔ The latest upstream changes (presumably #128534) made this pull request unmergeable. Please resolve the merge conflicts.

@tgross35
Copy link
Contributor Author

tgross35 commented Aug 6, 2024

Splitting this into #128722 and #128724 to update separately after the lockfile split.

This will no longer be needed as a fix for f128 symbols since I'm just removing reliance on cargo features rust-lang/compiler-builtins#652 / #128691.

@tgross35 tgross35 closed this Aug 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Compiler builtins missing f16/f128 symbols
7 participants