Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Spurious glibc Version Error When Using Build Scripts #724

Open
Alexhuszagh opened this issue May 27, 2022 · 11 comments
Open

Spurious glibc Version Error When Using Build Scripts #724

Alexhuszagh opened this issue May 27, 2022 · 11 comments

Comments

@Alexhuszagh
Copy link
Contributor

Alexhuszagh commented May 27, 2022

Issue Description

When using build scripts, even if rustc changes (path or target), cargo won't always do a full re-compile, and this can lead to spurious glibc errors. This is linked to rust-lang/cargo#10367 (comment). Since we change targets all the time, we're much more likely to cause this "feature" to manifest than standard builds.

There's 2 solutions:

  1. Use a custom target directory for each cross-compilation target (--target-dir path/to/x).
  2. Use cargo clean when the error appears.

More detailed information and a robust workaround are found on the wiki.

Update

This seems to happen sporadically, with any mixing of toolchains or targets. The issue occurs sporadically when the toolchain changes, and seems to be related to #858 (comment). The solution seems to be running cargo clean (with the appropriate target dir), but there needs to be a better solution.

Version Info

$ cross --version
cross 0.2.1 (459e03e 2022-05-26)
Warning: Falling back to `cargo` on the host.
cargo 1.63.0-nightly (39ad1039d 2022-05-25)

Sample code to reproduce:

$ git clone https://github.com/cross-rs/rust-cpp-hello-word
$ cd rust-cpp-hello-word
$ cross run --target powerpc64-unknown-linux-gnu --verbose
+ "rustc" "--print" "sysroot"
+ "rustup" "toolchain" "list"
+ "rustup" "target" "list" "--toolchain" "nightly-x86_64-unknown-linux-gnu"
+ "rustup" "component" "list" "--toolchain" "nightly-x86_64-unknown-linux-gnu"
+ "/usr/bin/docker" "run" "--userns" "host" "-e" "PKG_CONFIG_ALLOW_CROSS=1" "--rm" "--user" "1000:1000" "-e" "XARGO_HOME=/xargo" "-e" "CARGO_HOME=/cargo" "-e" "CARGO_TARGET_DIR=/target" "-e" "USER=ahuszagh" "-e" "CROSS_RUNNER=" "-v" "/home/ahuszagh/.xargo:/xargo:Z" "-v" "/home/ahuszagh/.cargo:/cargo:Z" "-v" "/cargo/bin" "-v" "/home/ahuszagh/Desktop/cross/rust-cpp-hello-word:/project:Z" "-v" "/home/ahuszagh/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu:/rust:Z,ro" "-v" "/home/ahuszagh/Desktop/cross/rust-cpp-hello-word/target:/target:Z" "-w" "/project" "-i" "-t" "ghcr.io/cross-rs/powerpc64-unknown-linux-gnu:main" "sh" "-c" "PATH=$PATH:/rust/bin cargo run --target powerpc64-unknown-linux-gnu --verbose"
       Fresh cc v1.0.73
   Compiling hellopp v0.1.0 (/project)
     Running `/target/debug/build/hellopp-0a565c6e09d5b0ce/build-script-build`
error: failed to run custom build command for `hellopp v0.1.0 (/project)`

Caused by:
  process didn't exit successfully: `/target/debug/build/hellopp-0a565c6e09d5b0ce/build-script-build` (exit status: 1)
  --- stderr
  /target/debug/build/hellopp-0a565c6e09d5b0ce/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.25' not found (required by /target/debug/build/hellopp-0a565c6e09d5b0ce/build-script-build)
  /target/debug/build/hellopp-0a565c6e09d5b0ce/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.27' not found (required by /target/debug/build/hellopp-0a565c6e09d5b0ce/build-script-build)

This should compile normally, however, we are unable to find the appropriate glibc version. This was detected while debugging #441.

@Alexhuszagh Alexhuszagh changed the title glibc version error in powerpc64-unknown-linux-gnu target glibc version error in nightly targets May 27, 2022
@Alexhuszagh Alexhuszagh removed the A-powerpc Area: PowerPC targets label May 27, 2022
@Emilgardis
Copy link
Member

I can't replicate this with nightly 2022-05-26 on windows with the latest powerpc64 image (sha256:d52b1c1e2f0666cf8e244e890f8e6dfe1907e427e394de10d11c385ae728912c)

❯ cross run --target powerpc64-unknown-linux-gnu --verbose
+ "cargo" "metadata" "--format-version=1" "--filter-platform" "powerpc64-unknown-linux-gnu" "--features" ""
+ "rustc" "--print" "sysroot"
+ "rustup" "toolchain" "list"
Warning: using newer rustc `1.63.0-nightly (490324f7b 2022-05-26)` for the target. Current active rustc on the host is `rustc 1.63.0-nightly (76761db59 2022-05-24)`.
 > Update with `rustup update`
+ "rustup" "target" "list" "--toolchain" "nightly-x86_64-unknown-linux-gnu"
+ "rustup" "component" "list" "--toolchain" "nightly-x86_64-unknown-linux-gnu"
+ "C:\\WINDOWS\\system32\\wsl.exe" "-e" "wslpath" "-a" "G:\\workspace\\small dev space\\cross-issue724\\rust-cpp-hello-word"
+ "C:\\WINDOWS\\system32\\wsl.exe" "-e" "wslpath" "-a" "G:\\workspace\\small dev space\\cross-issue724\\rust-cpp-hello-word"
+ "C:\\Program Files\\Docker\\Docker\\resources\\bin\\docker.EXE" "run" "--userns" "host" "-e" "PKG_CONFIG_ALLOW_CROSS=1" "--rm" "-e" "XARGO_HOME=/xargo" "-e" "CARGO_HOME=/cargo" "-e" "CARGO_TARGET_DIR=/target" "-e" "USER=Emil" "-e" "CROSS_RUNNER=" "-v" "C:\\Users\\Emil\\.xargo:/xargo:Z" "-v" "C:\\Users\\Emil\\.cargo:/cargo:Z" "-v" "/cargo/bin" "-v" "G:\\workspace\\small dev space\\cross-issue724\\rust-cpp-hello-word:/project:Z" "-v" "C:\\Users\\Emil\\.rustup\\toolchains\\nightly-x86_64-unknown-linux-gnu:/rust:Z,ro" "-v" "G:\\workspace\\small dev space\\cross-issue724\\rust-cpp-hello-word\\target:/target:Z" "-w" "/project" "-i" "-t" "ghcr.io/cross-rs/powerpc64-unknown-linux-gnu:main" "sh" "-c" "PATH=$PATH:/rust/bin cargo run --target powerpc64-unknown-linux-gnu --verbose"
   Compiling cc v1.0.73
     Running `rustc --crate-name cc --edition=2018 /cargo/registry/src/git.luolix.top-1ecc6299db9ec823/cc-1.0.73/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debuginfo=2 -C metadata=8451de8edbca495a -C extra-filename=-8451de8edbca495a --out-dir /target/debug/deps -L dependency=/target/debug/deps --cap-lints allow`
   Compiling hellopp v0.1.0 (/project)
     Running `rustc --crate-name build_script_build build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debuginfo=2 -C metadata=0a565c6e09d5b0ce -C extra-filename=-0a565c6e09d5b0ce --out-dir /target/debug/build/hellopp-0a565c6e09d5b0ce -C incremental=/target/debug/incremental -L dependency=/target/debug/deps --extern cc=/target/debug/deps/libcc-8451de8edbca495a.rlib`
     Running `/target/debug/build/hellopp-0a565c6e09d5b0ce/build-script-build`
     Running `rustc --crate-name hellopp src/main.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debuginfo=2 -C metadata=fc5c3be60a3d8fee -C extra-filename=-fc5c3be60a3d8fee --out-dir /target/powerpc64-unknown-linux-gnu/debug/deps --target powerpc64-unknown-linux-gnu -C linker=powerpc64-linux-gnu-gcc -C incremental=/target/powerpc64-unknown-linux-gnu/debug/incremental -L dependency=/target/powerpc64-unknown-linux-gnu/debug/deps -L dependency=/target/debug/deps -L native=/target/powerpc64-unknown-linux-gnu/debug/build/hellopp-618833d9398f1fb8/out -l static=hellopp -l stdc++`
    Finished dev [unoptimized + debuginfo] target(s) in 30.29s
     Running `/linux-runner powerpc64 /target/powerpc64-unknown-linux-gnu/debug/hellopp`
Hello, world!

@Alexhuszagh
Copy link
Contributor Author

Running cargo clean fixed this, so it might be a build fingerprint issue or incremental build issue. Either way, not replicating for me anymore.

@Alexhuszagh
Copy link
Contributor Author

I'm getting this when I mix the nightly and stable toolchains for a given target.

$ cross run --target armv7-unknown-linux-gnueabihf
    Finished dev [unoptimized + debuginfo] target(s) in 0.11s
     Running `/linux-runner armv7 /target/armv7-unknown-linux-gnueabihf/debug/hellopp`
Hello, world!
$ cross run +nightly --target armv7-unknown-linux-gnueabihf
Warning: using newer rustc `1.63.0-nightly (490324f7b 2022-05-26)` for the target. Current active rustc on the host is `rustc 1.61.0 (fe5b13d68 2022-05-18)`.
 > Update with `rustup update`
   Compiling hellopp v0.1.0 (/project)
error: failed to run custom build command for `hellopp v0.1.0 (/project)`

Caused by:
  process didn't exit successfully: `/target/debug/build/hellopp-0a565c6e09d5b0ce/build-script-build` (exit status: 1)
  --- stderr
  /target/debug/build/hellopp-0a565c6e09d5b0ce/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.29' not found (required by /target/debug/build/hellopp-0a565c6e09d5b0ce/build-script-build)
  /target/debug/build/hellopp-0a565c6e09d5b0ce/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by /target/debug/build/hellopp-0a565c6e09d5b0ce/build-script-build)
  /target/debug/build/hellopp-0a565c6e09d5b0ce/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.25' not found (required by /target/debug/build/hellopp-0a565c6e09d5b0ce/build-script-build)
  /target/debug/build/hellopp-0a565c6e09d5b0ce/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.27' not found (required by /target/debug/build/hellopp-0a565c6e09d5b0ce/build-script-build)

We might have to document this as a known issue. Thoughts @Emilgardis?

@Emilgardis
Copy link
Member

I want to know why this happens. Is the build script not invalidated with different toolchains?

@Alexhuszagh
Copy link
Contributor Author

Still not sure, it doesn't reproduce very frequently. I think it's the above, but it should be invalidated and most of the time it is.

@Emilgardis Emilgardis reopened this Jun 6, 2022
@Emilgardis Emilgardis changed the title glibc version error in nightly targets spurious glibc version error in build script build Jun 6, 2022
@Alexhuszagh Alexhuszagh added the A-glibc Area: glibc targets label Jun 22, 2022
Alexhuszagh added a commit to Alexhuszagh/cross that referenced this issue Jun 22, 2022
Required to patch cross-rs#724 without deleting the entire volume for persistent
data volumes. A few changes were required: the entire `/cross` mount
prefix must be owned by the user so `/cross/target` can be removed.
Next, we use the full path to the mounted target directory, rather than
the symlink, since `cargo clean` would just delete the symlink. Finally,
we've added `cargo clean` to a list of known subcommands, and it only
needs docker if we use a remote host.
@Emilgardis Emilgardis pinned this issue Jun 22, 2022
Alexhuszagh added a commit to Alexhuszagh/cross that referenced this issue Jun 22, 2022
Required to patch cross-rs#724 without deleting the entire volume for persistent
data volumes. A few changes were required: the entire `/cross` mount
prefix must be owned by the user so `/cross/target` can be removed.
Next, we use the full path to the mounted target directory, rather than
the symlink, since `cargo clean` would just delete the symlink. Finally,
we've added `cargo clean` to a list of known subcommands, and it only
needs docker if we use a remote host.
Emilgardis pushed a commit to Emilgardis/cross that referenced this issue Jun 24, 2022
Required to patch cross-rs#724 without deleting the entire volume for persistent
data volumes. A few changes were required: the entire `/cross` mount
prefix must be owned by the user so `/cross/target` can be removed.
Next, we use the full path to the mounted target directory, rather than
the symlink, since `cargo clean` would just delete the symlink. Finally,
we've added `cargo clean` to a list of known subcommands, and it only
needs docker if we use a remote host.
Darksonn added a commit to tokio-rs/tokio that referenced this issue Jun 29, 2022
CI is running into cross-rs/cross#724, which
should be fixed if we clear the cache.
Darksonn added a commit to tokio-rs/tokio that referenced this issue Jun 29, 2022
CI is running into cross-rs/cross#724, which
should be fixed if we clear the cache.
@Emilgardis
Copy link
Member

Been trying to make a consistent reproduction of this but nothing seems to work to reproduce it. I think what we need to do is identify exactly when this happens, and then if there are any other workarounds aside from cargo clean, I still suspect -Zbinary-dep-depinfo would solve this.

I've been trying to reproduce it with cargo build and then cross build, but nothing reproduces it in the build script (I have been able to reproduce it without a build-script, but that's another case). See https://github.com/Emilgardis/cross-issue724/actions/workflows/ci.yml

@Imyr
Copy link

Imyr commented Mar 26, 2023

In my particular case, it happens when cross build --release is used.
just cross build gives no such error.
Thought it would help.

Status: Downloaded newer image for ghcr.io/cross-rs/aarch64-unknown-linux-gnu:main
   Compiling libc v0.2.139
error: failed to run custom build command for `libc v0.2.139`

Caused by:
  process didn't exit successfully: `/target/release/build/libc-4e3b2cf9f91fdbab/build-script-build` (exit status: 1)
  --- stderr
  /target/release/build/libc-4e3b2cf9f91fdbab/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /target/release/build/libc-4e3b2cf9f91fdbab/build-script-build)
  /target/release/build/libc-4e3b2cf9f91fdbab/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /target/release/build/libc-4e3b2cf9f91fdbab/build-script-build)
  /target/release/build/libc-4e3b2cf9f91fdbab/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /target/release/build/libc-4e3b2cf9f91fdbab/build-script-build)
warning: build failed, waiting for other jobs to finish...
Error: Process completed with exit code 101.

Running cross on Github Actions, snippet attached;

         cargo build --release &&
         mv target/release/bitnet target/release/bitnet-linux-gnu-x86_64 &&
         cargo install cross --git https://github.com/cross-rs/cross &&
         sudo apt-get install qemu binfmt-support qemu-user-static &&
         docker run --privileged --rm tonistiigi/binfmt --install all &&
         rustup target install aarch64-unknown-linux-gnu &&
         cross build --target aarch64-unknown-linux-gnu &&
         mv target/aarch64-unknown-linux-gnu/debug/bitnet target/release/bitnet-linux-gnu-aarch64 &&
         rustup target install aarch64-linux-android &&
         cross build --target aarch64-linux-android &&
         mv target/aarch64-linux-android/debug/bitnet target/release/bitnet-linux-android-aarch64

Dependencies in Cargo.toml file;

[dependencies]
reqwest = "0.11.13"
html-escape = "0.2.13"
tokio = { version = "1", features = ["macros"] }
clap = { version = "4.1.4", features = ["derive"] }
openssl = { version = "0.10.45", features = ["vendored"] }
git2 = {version = "0.13.22", features = ["vendored-libgit2"]}

EDIT:

        mkdir release &&
        cargo build --release &&
        mv target/release/bitnet release/bitnet-linux-gnu-x86_64 &&
        cargo install cross --git https://github.com/cross-rs/cross &&
        sudo apt-get install qemu binfmt-support qemu-user-static &&
        docker run --privileged --rm tonistiigi/binfmt --install all &&
        cargo clean &&
        rustup target install aarch64-unknown-linux-gnu &&
        cross build --release --target aarch64-unknown-linux-gnu &&
        mv target/aarch64-unknown-linux-gnu/release/bitnet release/bitnet-linux-gnu-aarch64 &&
        cargo clean &&
        rustup target install aarch64-linux-android &&
        cross build --release --target aarch64-linux-android &&
        mv target/aarch64-linux-android/release/bitnet release/bitnet-linux-android-aarch64

using cargo clean as suggested worked.

@WillBuik
Copy link

WillBuik commented Jun 3, 2023

I'm not certain it is the same issue but I have a consistent and simple way to repro something like this
with cross build --release. Running cargo clean doesn't work around it, you need to use
separate target directories.

Steps:

  1. Create a new binary crate with chrono v0.4.26 as a dependency.
  2. run cross build --release --target x86_64-unknown-linux-musl
  3. run cross build --release --target aarch64-unknown-linux-musl

The second cross build will fail with:

error: failed to run custom build command for `num-traits v0.2.15`

Caused by:
  process didn't exit successfully: `/target/release/build/num-traits-60547cd1ad1882f5/build-script-build` (exit status: 1)
  --- stderr
  /target/release/build/num-traits-60547cd1ad1882f5/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.29' not found (required by /target/release/build/num-traits-60547cd1ad1882f5/build-script-build)
  /target/release/build/num-traits-60547cd1ad1882f5/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by /target/release/build/num-traits-60547cd1ad1882f5/build-script-build)

I am running this on an Intel Mac with Docker Desktop with rustc 1.72.0-nightly (871b59520 2023-05-31) on the host.

lu-zero added a commit to sifis-home/third-party-application that referenced this issue Jun 14, 2023
Prevent cross from considering the targets built for x86_64 fresh for
aarch64.

See: cross-rs/cross#724
newtykip added a commit to newtykip/johnny that referenced this issue Jul 17, 2023
@denfren
Copy link

denfren commented Jul 17, 2023

I noticed that the order seems to matter. Building aarch64 first consistently works for me while building x86_64 first consistently fails. This is when compiling after cargo clean...results get random once anything was compiled.

FlareFlo added a commit to Warthunder-Open-Source-Foundation/wt_ext_cli that referenced this issue Oct 9, 2023
marchaen added a commit to marchaen/another-keyboard-layer that referenced this issue Oct 27, 2023
Building the whole project with `dotnet publish -c Release --os win`
would fail with the message "glibc version 2.25 not found" this was
related to caching problems while compiling the build script and a
workaround is specifying a different target directory for each target
that should be build.

See also the following issues for more details about the problem:

- cross-rs/cross#988
- cross-rs/cross#724
marchaen added a commit to marchaen/another-keyboard-layer that referenced this issue Oct 27, 2023
Unfortunately the follwing problem was reintroduced because the linux
target was removed without checking if debug builds still worked.

Issue: cross-rs/cross#724
gj added a commit to jdeff/oso that referenced this issue Jan 12, 2024
Seeing if
[this](cross-rs/cross#724 (comment))
is a quick fix so I can avoid any deeper debugging.
gj added a commit to osohq/oso that referenced this issue Jan 13, 2024
* Update nokogiri to version with support for apple silicon

* Update gem dependencies to address spec failures running with Ruby 3

* Require ActiveRecordAdapter from spec that uses it

* Add support for both x86_64 and arm64 linux architectures

* Build both x86 and arm dynamic libraries for ruby

* Use Ruby 2.6 as minimum tested version and 3.2 for all ruby-related tasks

Rationale: 2.6.10 is what ships with macOS 14.0 even though support for it
stopped in March of 2022 and 3.2 is the latest stable version.

* Ensure the ruby libs are copied and available for testing

* Bump solargraph and rubocop

Disable new cops for existing violations

* Update bundler to get rid of SPELL_CHECKERS warning

* Do not suggest extensions

* Simplify YARD docs for query method

Add the host and bindings parameters so calls that contain them don't fail typecheck

* Must use Ruby 2.4 in order to build the docs on main

* Must use Ruby 2.4 in order to build the docs on main

* Use .ruby-version instead of hard-coding the current ruby version in Github actions

The rationale for this is from the way that docs diffing works. Both the base branch and the new branch need to be able to build using the version of Ruby supported by that branch. Without this change, when a PR is introduced that updates the target Ruby version, a followup PR will be required to update the github action for building the 'base' branch's version of the docs. With this change, github actions will choose the version of Ruby targeted by the branch it is trying to build

* Clippy

* Cargo fmt

* Test

* Try building aarch64 first

Seeing if
[this](cross-rs/cross#724 (comment))
is a quick fix so I can avoid any deeper debugging.

* Keep shuffling the deck chairs

* Revert "Test"

This reverts commit 7feebd1.

* Add 0.27.3 changelog

* Bump version

* Let's just release now instead

---------

Co-authored-by: Gabe Jackson <17556281+gj@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants