cargo test --doc from make -j always triggers "failed to connect to jobserver from environment"

[Deewiant]

Problem

Having a Makefile run cargo test --doc, even with the command flagged as recursive with + so that make -j enables the jobserver for it, always triggers the rustc warning about a jobserver issue ("failed to connect to jobserver from environment variable"). I expect this to not happen.

I'm not sure what the "actual" impact beyond the warning is, since at least compiling dependency crates seems to follow the -j as expected. Maybe it only affects the compilation step for the final doctest binaries?

I guessed that this is most likely a cargo issue, although the interaction among cargo/rustdoc/rustc for doctests isn't very clear to me so maybe the root cause lies somewhere else.

Steps

1. cargo new --lib foo && cd foo && printf 'doctests:\n\t+cargo test --doc' | make -j2 -f -

Results on my system (GNU Make 4.3, cargo 1.80.1):

    Creating library `foo` package
note: see more `Cargo.toml` keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
cargo test --doc
   Compiling foo v0.1.0 (/tmp/foo)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.19s
   Doc-tests foo
warning: failed to connect to jobserver from environment variable `MAKEFLAGS=" -j2 --jobserver-auth=3,4"`: cannot open file descriptor 3 from the jobserver environment variable value: Bad file descriptor (os error 9)
  |
  = note: the build environment is likely misconfigured


running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Changing --doc to --lib does not trigger the issue:

    Creating library `foo` package
note: see more `Cargo.toml` keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
cargo test --lib
   Compiling foo v0.1.0 (/tmp/foo)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.45s
     Running unittests src/lib.rs (target/debug/deps/foo-8236ccd36d2d63e2)

running 1 test
test tests::it_works ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

And removing --doc triggers it only during the doctests stage:

    Creating library `foo` package
note: see more `Cargo.toml` keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
cargo test
   Compiling foo v0.1.0 (/tmp/foo)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.43s
     Running unittests src/lib.rs (target/debug/deps/foo-8236ccd36d2d63e2)

running 1 test
test tests::it_works ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

   Doc-tests foo
warning: failed to connect to jobserver from environment variable `MAKEFLAGS=" -j2 --jobserver-auth=3,4"`: cannot open file descriptor 3 from the jobserver environment variable value: Bad file descriptor (os error 9)
  |
  = note: the build environment is likely misconfigured


running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Possible Solution(s)

No response

Notes

The only workaround I've come up with is to use MAKEFLAGS= cargo test --doc, but that of course causes the cargo test --doc invocation to use more parallelism than given to make -j, potentially oversubscribing the CPU. cargo test --doc -j1 can then be used as an additional workaround, with the natural downside of slowing down compilation a lot.

Version

cargo 1.80.1 (376290515 2024-07-16)
release: 1.80.1
commit-hash: 37629051518c3df9ac2c1744589362a02ecafa99
commit-date: 2024-07-16
host: x86_64-unknown-linux-gnu
libgit2: 1.7.2 (sys:0.18.3 vendored)
libcurl: 8.6.0-DEV (sys:0.4.72+curl-8.6.0 vendored ssl:OpenSSL/1.1.1w)
ssl: OpenSSL 1.1.1w  11 Sep 2023
os: Ubuntu 22.4.0 (jammy) [64-bit]

[weihanglo]

Thanks for the report and the reproducible step!

This doesn't seem to be reproduced on the latest nightly cargo 1.82.0-nightly (2f738d617 2024-08-13), but still happens on beta cargo 1.81.0-beta.5 (ca2346bd3 2024-07-31). It seems like rust-lang/jobserver-rs#99, which has been fixed by rust-lang/jobserver-rs#100 and released in jobserver@0.1.32.

Checked out to beta Cargo and update jobserver to 0.1.32 and the warning is gone.
The warning is also gone when using GNU Make 4.4+, which recognizes the new fifo:PATH auth style.

So, I believe the mysterious warning is no longer there on nightly.

That said, I am unsure whether doctest respects jobserver at this moment. We may need to call process.inherit_jobserver(client) for doctest processes, though that needs some investigation and should be in its own separate issue.

cargo/src/cargo/ops/cargo_test.rs

Line 213 in d585829

let mut p = compilation.rustdoc_process(unit, *script_meta)?;

---

Anyhow, could you help confirm that on nightly the warning no longer shows up?

[Deewiant]

Yep, I can confirm that with cargo +beta test --doc the issue is still present while with cargo +nightly test --doc it's gone!

[weihanglo]

Sounds great! Close as resolved :)