"memory allocation of .. bytes failed" appears when using cargo run test to run specific test name

[chentodd]

Problem

cargo run test shows "memory allocation of .. bytes failed" when I run specific test case:

Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.20s
     Running `target/debug/cargo test source_replacement`
memory allocation of 63063237072 bytes failed
Aborted (core dumped)

I expect there are no issues when I use cargo run test to run the tests.

Steps

1. Run rustup update to update needed tools to build cargo
2. Clone cargo repository
3. cd into cargo repository
4. Change to master branch
5. Run cargo check --workspace
6. Run cargo build
7. Run cargo run test source_replacement

here is the test video that contains step.5 to step.7 for the reference: 20241021_cargo_test_video.zip

Possible Solution(s)

No response

Notes

The error message in this issue is similar to #7543

Version

cargo 1.84.0 (edd36eb 2024-10-21)
release: 1.84.0
commit-hash: edd36eb
commit-date: 2024-10-21
host: x86_64-unknown-linux-gnu
libgit2: 1.8.1 (sys:0.19.0 vendored)
libcurl: 8.5.0 (sys:0.4.74+curl-8.9.0 system ssl:OpenSSL/3.0.13)
os: Ubuntu 24.4.0 (noble) [64-bit]

additional hardware information of test machine:
Memory: 16 GB

[epage]

I wasn't able to reproduce. Unsure what the difference is.

[weihanglo]

The RAM size is critical.

additional hardware information of test machine:
Memory: 16 GB

I reproduced it on a machine with 4 GiB RAM.

(24-10-24 17:12:24) <0> [~/cargo]  
%  cargo -vV
cargo 1.82.0 (8f40fc59f 2024-08-21)
release: 1.82.0
commit-hash: 8f40fc59fb0c8df91c97405785197f3c630304ea
commit-date: 2024-08-21
host: x86_64-unknown-linux-gnu
libgit2: 1.8.1 (sys:0.19.0 vendored)
libcurl: 8.9.0-DEV (sys:0.4.74+curl-8.9.0 vendored ssl:OpenSSL/1.1.1w)
ssl: OpenSSL 1.1.1w  11 Sep 2023
os: Amazon Linux AMI 2.0.0 [64-bit]

(24-10-24 17:12:30) <0> [~/cargo]  
%  free -h
              total        used        free      shared  buff/cache   available
Mem:           3.6G        520M        1.6G        1.7M        1.5G        2.8G
Swap:            0B          0B          0B

(24-10-24 17:12:35) <0> [~/cargo]  
%  cargo run -F all-static -- test -F all-static source_replacement
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.23s
     Running `target/debug/cargo test -F all-static source_replacement`
memory allocation of 63063237072 bytes failed
zsh: abort      cargo run -F all-static -- test -F all-static source_replacement

Testing other combinations:

• ✅ cargo test -F all-static source_replacement
• ✅ cargo build -F all-static source_replacement
• ✅ ./target/debug/cargo build -F all-static
• ✅ ./target/release/cargo build -F all-static
• ❌ ./target/debug/cargo test -F all-static
• ❌ ./target/release/cargo test -F all-static
• ✅ cp ./target/debug/cargo dargo && cargo clean && ./dargo test -F all-static
• ✅ cp ./target/release/cargo rargo && cargo clean && ./rargo test -F all-static
• ✅ cp ./target/debug/cargo dargo && ./dargo build -F all-static
• ✅ cp ./target/debug/cargo dargo && ./dargo build -F all-static
• ❌ cp ./target/debug/cargo dargo && ./dargo test -F all-static
• ❌ cp ./target/debug/cargo dargo && ./dargo test -F all-static

So it looks like the requirements are

• With existing build cache
• Using the newly built cargo binary
• Running cargo test

to trigger a core dump.

I may investigate it more when having time. At least we need to be sure that it is really an edge case for cargo itself.

[weihanglo]

Did a cargo t against rust-lang/crates.io with both debug/release built cargo from master. It works alright with 4GiB RAM (c5.large AWS EC2 instance). So, probably this is probably a Cargo-specific issue.

Labeled as low priority though feel free to help debug it!

[AlexTMjugador]

I got hit by this bug when running cargo check on this commit (cargo 1.84.0-nightly (cf53cc54b 2024-10-18), rustc 1.84.0-nightly (c1db4dc24 2024-10-25)). My box is also x86_64-unknown-linux-gnu, but it has 32 GiB of RAM, so throwing more RAM at the problem does not help either. I didn't have to set RUST_BACKTRACE to any value.

Edit: doing a cargo clean fixed it. Looks like doing a rustup update while having an older Rust version build cache is troublesome, is this a supported use case or does it happen to mostly work in practice in spite of it not being supported? For what it's worth, it's not the first time I do a rustup update with an already existing build cache.

[linyihai]

Edit: doing a cargo clean fixed it. Looks like doing a rustup update while having an older Rust version build cache is troublesome, is this a supported use case or does it happen to mostly work in practice in spite of it not being supported? For what it's worth, it's not the first time I do a rustup update with an already existing build cache.

I encountered this problem recently. And I did cargo clean and rustup update to workaround it.

[weihanglo]

so throwing more RAM at the problem does not help either

It does help if you have more than 63063237072 bytes available in your RAM 😆.
I do have a machine with 128 GiB RAM and can't reproduce this issue on it.

---

Since there is a non Cargo project affected, I removed the P-low label.

[weihanglo]

I compiled master Cargo (e75214e) with -Zoom=panic and got this backtrace.

RUSTC_BOOTSTRAP=1 RUSTFLAGS=-Zoom=panic cargo b -F all-static
RUST_BACKTRACE=1 ./target/debug/cargo t

Backtrace

thread 'main' panicked at std/src/alloc.rs:355:9:
memory allocation of 63063237072 bytes failed
stack backtrace:
   0: rust_begin_unwind
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/std/src/panicking.rs:665:5
   1: core::panicking::panic_fmt
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/core/src/panicking.rs:74:14
   2: std::alloc::default_alloc_error_hook
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/std/src/alloc.rs:355:9
   3: std::alloc::rust_oom
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/std/src/alloc.rs:375:5
   4: __rg_oom
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/std/src/alloc.rs:371:1
   5: alloc::alloc::handle_alloc_error::rt_error
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/alloc/src/alloc.rs:403:13
   6: alloc::alloc::handle_alloc_error
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/alloc/src/alloc.rs:409:9
   7: alloc::raw_vec::handle_error
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/alloc/src/raw_vec.rs:793:38
   8: alloc::raw_vec::RawVecInner<A>::with_capacity_in
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/alloc/src/raw_vec.rs:431:25
   9: alloc::raw_vec::RawVec<T,A>::with_capacity_in
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/alloc/src/raw_vec.rs:194:20
  10: alloc::vec::Vec<T,A>::with_capacity_in
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/alloc/src/vec/mod.rs:803:20
  11: alloc::vec::Vec<T>::with_capacity
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/alloc/src/vec/mod.rs:483:9
  12: cargo::core::compiler::fingerprint::EncodedDepInfo::parse
             at ./src/cargo/core/compiler/fingerprint/mod.rs:2272:23
  13: cargo::core::compiler::fingerprint::parse_dep_info
             at ./src/cargo/core/compiler/fingerprint/mod.rs:1928:22
  14: cargo::core::compiler::fingerprint::LocalFingerprint::find_stale_item
             at ./src/cargo/core/compiler/fingerprint/mod.rs:835:34
  15: cargo::core::compiler::fingerprint::Fingerprint::check_filesystem
             at ./src/cargo/core/compiler/fingerprint/mod.rs:1236:33
  16: cargo::core::compiler::fingerprint::calculate
             at ./src/cargo/core/compiler/fingerprint/mod.rs:1432:5
  17: cargo::core::compiler::fingerprint::DepFingerprint::new
             at ./src/cargo/core/compiler/fingerprint/mod.rs:1315:27
  18: cargo::core::compiler::fingerprint::calculate_normal::{{closure}}
             at ./src/cargo/core/compiler/fingerprint/mod.rs:1467:24
  19: core::iter::adapters::map::map_try_fold::{{closure}}
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/core/src/iter/adapters/map.rs:95:28
  20: core::iter::adapters::filter::filter_try_fold::{{closure}}
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/core/src/iter/adapters/filter.rs:86:44
  21: <alloc::vec::into_iter::IntoIter<T,A> as core::iter::traits::iterator::Iterator>::try_fold
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/alloc/src/vec/into_iter.rs:346:25
  22: <core::iter::adapters::filter::Filter<I,P> as core::iter::traits::iterator::Iterator>::try_fold
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/core/src/iter/adapters/filter.rs:151:9
  23: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::try_fold
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/core/src/iter/adapters/map.rs:121:9
  24: <core::iter::adapters::GenericShunt<I,R> as core::iter::traits::iterator::Iterator>::try_fold
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/core/src/iter/adapters/mod.rs:191:9
  25: <I as alloc::vec::in_place_collect::SpecInPlaceCollect<T,I>>::collect_in_place
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/alloc/src/vec/in_place_collect.rs:380:13
  26: alloc::vec::in_place_collect::from_iter_in_place
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/alloc/src/vec/in_place_collect.rs:271:9
  27: alloc::vec::in_place_collect::from_iter_in_place{{reify.shim}}
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/alloc/src/vec/in_place_collect.rs:251:1
  28: alloc::vec::in_place_collect::<impl alloc::vec::spec_from_iter::SpecFromIter<T,I> for alloc::vec::Vec<T>>::from_iter
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/alloc/src/vec/in_place_collect.rs:246:9
  29: <alloc::vec::Vec<T> as core::iter::traits::collect::FromIterator<T>>::from_iter
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/alloc/src/vec/mod.rs:3410:9
  30: core::iter::traits::iterator::Iterator::collect
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/core/src/iter/traits/iterator.rs:2001:9
  31: <core::result::Result<V,E> as core::iter::traits::collect::FromIterator<core::result::Result<A,E>>>::from_iter::{{closure}}
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/core/src/result.rs:1979:51
  32: core::iter::adapters::try_process
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/core/src/iter/adapters/mod.rs:160:17
  33: <core::result::Result<V,E> as core::iter::traits::collect::FromIterator<core::result::Result<A,E>>>::from_iter
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/core/src/result.rs:1979:9
  34: core::iter::traits::iterator::Iterator::collect
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/core/src/iter/traits/iterator.rs:2001:9
  35: cargo::core::compiler::fingerprint::calculate_normal
             at ./src/cargo/core/compiler/fingerprint/mod.rs:1464:24
  36: cargo::core::compiler::fingerprint::calculate
             at ./src/cargo/core/compiler/fingerprint/mod.rs:1425:9
  37: cargo::core::compiler::fingerprint::DepFingerprint::new
             at ./src/cargo/core/compiler/fingerprint/mod.rs:1315:27
  38: cargo::core::compiler::fingerprint::calculate_normal::{{closure}}
             at ./src/cargo/core/compiler/fingerprint/mod.rs:1467:24
  39: core::iter::adapters::map::map_try_fold::{{closure}}
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/core/src/iter/adapters/map.rs:95:28
  40: core::iter::adapters::filter::filter_try_fold::{{closure}}
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/core/src/iter/adapters/filter.rs:86:44
  41: <alloc::vec::into_iter::IntoIter<T,A> as core::iter::traits::iterator::Iterator>::try_fold
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/alloc/src/vec/into_iter.rs:346:25
  42: <core::iter::adapters::filter::Filter<I,P> as core::iter::traits::iterator::Iterator>::try_fold
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/core/src/iter/adapters/filter.rs:151:9
  43: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::try_fold
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/core/src/iter/adapters/map.rs:121:9
  44: <core::iter::adapters::GenericShunt<I,R> as core::iter::traits::iterator::Iterator>::try_fold
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/core/src/iter/adapters/mod.rs:191:9
  45: <I as alloc::vec::in_place_collect::SpecInPlaceCollect<T,I>>::collect_in_place
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/alloc/src/vec/in_place_collect.rs:380:13
  46: alloc::vec::in_place_collect::from_iter_in_place
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/alloc/src/vec/in_place_collect.rs:271:9
  47: alloc::vec::in_place_collect::from_iter_in_place{{reify.shim}}
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/alloc/src/vec/in_place_collect.rs:251:1
  48: alloc::vec::in_place_collect::<impl alloc::vec::spec_from_iter::SpecFromIter<T,I> for alloc::vec::Vec<T>>::from_iter
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/alloc/src/vec/in_place_collect.rs:246:9
  49: <alloc::vec::Vec<T> as core::iter::traits::collect::FromIterator<T>>::from_iter
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/alloc/src/vec/mod.rs:3410:9
  50: core::iter::traits::iterator::Iterator::collect
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/core/src/iter/traits/iterator.rs:2001:9
  51: <core::result::Result<V,E> as core::iter::traits::collect::FromIterator<core::result::Result<A,E>>>::from_iter::{{closure}}
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/core/src/result.rs:1979:51
  52: core::iter::adapters::try_process
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/core/src/iter/adapters/mod.rs:160:17
  53: <core::result::Result<V,E> as core::iter::traits::collect::FromIterator<core::result::Result<A,E>>>::from_iter
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/core/src/result.rs:1979:9
  54: core::iter::traits::iterator::Iterator::collect
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/core/src/iter/traits/iterator.rs:2001:9
  55: cargo::core::compiler::fingerprint::calculate_normal
             at ./src/cargo/core/compiler/fingerprint/mod.rs:1464:24
  56: cargo::core::compiler::fingerprint::calculate
             at ./src/cargo/core/compiler/fingerprint/mod.rs:1425:9
  57: cargo::core::compiler::fingerprint::prepare_target
             at ./src/cargo/core/compiler/fingerprint/mod.rs:428:23
  58: cargo::core::compiler::compile
             at ./src/cargo/core/compiler/mod.rs:194:23
  59: cargo::core::compiler::build_runner::BuildRunner::compile
             at ./src/cargo/core/compiler/build_runner/mod.rs:192:13
  60: cargo::ops::cargo_compile::compile_ws
             at ./src/cargo/ops/cargo_compile/mod.rs:162:9
  61: cargo::ops::cargo_compile::compile_with_exec
             at ./src/cargo/ops/cargo_compile/mod.rs:141:5
  62: cargo::ops::cargo_compile::compile
             at ./src/cargo/ops/cargo_compile/mod.rs:128:5
  63: cargo::ops::cargo_test::compile_tests
             at ./src/cargo/ops/cargo_test.rs:105:27
  64: cargo::ops::cargo_test::run_tests
             at ./src/cargo/ops/cargo_test.rs:68:23
  65: cargo::commands::test::exec
             at ./src/bin/cargo/commands/test.rs:114:5
  66: cargo::cli::Exec::exec
             at ./src/bin/cargo/cli.rs:474:36
  67: cargo::cli::main
             at ./src/bin/cargo/cli.rs:96:9
  68: cargo::main
             at ./src/bin/cargo/main.rs:57:9
  69: core::ops::function::FnOnce::call_once
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

It was this dep info file target/debug/.fingerprint/cargo-test-macro-3b2e77deed00ae97/dep-lib-cargo_test_macro during decoding, it got checksum=1, and thought it had to support checksum freshness.

cargo/src/cargo/core/compiler/fingerprint/mod.rs

Line 2259 in e75214e

let checksum_info = has_checksum

As a result, Cargo then wrongly decoded nenv as "1313817439", and then pushed nenv number of (String, Option<String>) to the Vec which caused OOM-killed.

cargo/src/cargo/core/compiler/fingerprint/mod.rs

Lines 2271 to 2273 in e75214e

	let nenv = read_usize(bytes)?;
	let mut env = Vec::with_capacity(nenv);
	for _ in 0..nenv {

Feel like decoding a bool also for old dep-info is too fragile, though I may not be available in the next few hours to come up with a fix.

[weihanglo]

I got hit by this bug when running cargo check on this commit (cargo 1.84.0-nightly (cf53cc54b 2024-10-18), rustc 1.84.0-nightly (c1db4dc24 2024-10-25)). My box is also x86_64-unknown-linux-gnu, but it has 32 GiB of RAM, so throwing more RAM at the problem does not help either. I didn't have to set RUST_BACKTRACE to any value.

Edit: doing a cargo clean fixed it. Looks like doing a rustup update while having an older Rust version build cache is troublesome, is this a supported use case or does it happen to mostly work in practice in spite of it not being supported? For what it's worth, it's not the first time I do a rustup update with an already existing build cache.

@AlexTMjugador would you mind sharing steps to reproduce? Like which version of Cargo you used for the first build, and which cargo you use that encountered a core-dumped.

I have a hacky fix at hand, but unsure if it is worthy. Fingerprints generally are not shared between different toolchains. Unless you manually build cargo or have a pretty specialized toolchain setup, I still feel like normal users won't be bitten by this.

[AlexTMjugador]

@AlexTMjugador would you mind sharing steps to reproduce? Like which version of Cargo you used for the first build, and which cargo you use that encountered a core-dumped.

I have a hacky fix at hand, but unsure if it is worthy. Fingerprints generally are not shared between different toolchains. Unless you manually build cargo or have a pretty specialized toolchain setup, I still feel like normal users won't be bitten by this.

Sure! Please find the exact steps to reproduce this issue attached below. To ensure a clean and consistent environment, I used a temporary Docker container based on the rustlang/rust:nightly image, which currently refers to this digest.

Terminal session of reproduction steps

$ docker run -it --rm rustlang/rust:nightly # Digest: https://hub.docker.com/layers/rustlang/rust/nightly/images/sha256-eceb4b2464041d21977d3699e4a085e0859a13c2c5227839eb7e9219f53f26d6?context=explore
root@3fde24c67f0d:/# rustc --version # The rustc version used by default
rustc 1.84.0-nightly (df4ca44d3 2024-10-27)
root@3fde24c67f0d:/# rustup toolchain install nightly-2024-09-15 # The previous rustc version, chosen arbitrarily to be approximately as old as I remember having before updating
info: syncing channel updates for 'nightly-2024-09-15-x86_64-unknown-linux-gnu'
info: latest update on 2024-09-15, rust version 1.83.0-nightly (9b72238eb 2024-09-14)
info: downloading component 'cargo'
  8.5 MiB /   8.5 MiB (100 %)   2.6 MiB/s in  1s ETA:  0s
info: downloading component 'rust-std'
 28.1 MiB /  28.1 MiB (100 %)   5.6 MiB/s in  5s ETA:  0s
info: downloading component 'rustc'
 72.1 MiB /  72.1 MiB (100 %) 758.2 KiB/s in 32s ETA:  0s    
info: installing component 'cargo'
info: installing component 'rust-std'
 28.1 MiB /  28.1 MiB (100 %)  21.1 MiB/s in  1s ETA:  0s
info: installing component 'rustc'
 72.1 MiB /  72.1 MiB (100 %)  23.0 MiB/s in  3s ETA:  0s

  nightly-2024-09-15-x86_64-unknown-linux-gnu installed - rustc 1.83.0-nightly (9b72238eb 2024-09-14)

info: checking for self-update
root@3fde24c67f0d:/# cd tmp
root@3fde24c67f0d:/tmp# git clone https://github.com/ComunidadAylas/PackSquash.git
Cloning into 'PackSquash'...
remote: Enumerating objects: 20748, done.
remote: Counting objects: 100% (517/517), done.
remote: Compressing objects: 100% (168/168), done.
remote: Total 20748 (delta 302), reused 471 (delta 277), pack-reused 20231 (from 1)
Receiving objects: 100% (20748/20748), 6.74 MiB | 15.57 MiB/s, done.
Resolving deltas: 100% (13003/13003), done.
root@3fde24c67f0d:/tmp# cd PackSquash/
root@3fde24c67f0d:/tmp/PackSquash# git reset --hard f973bd3
HEAD is now at f973bd38 Revert "chore: ignore broken `clippy::needless_return` lint in latest nightlies"
root@3fde24c67f0d:/tmp/PackSquash# cargo +nightly-2024-09-15 check # Uses an older nightly toolchain
<Normal cargo check output, works fine>
root@3fde24c67f0d:/tmp/PackSquash# cargo check # Uses a recent nightly toolchain
memory allocation of 67045949232 bytes failed
Aborted (core dumped)

[weihanglo]

Thanks @AlexTMjugador for the reproduction! It was really helpful :)

Confirmed that this issue is an incompatibility bug introduced by -Zchecksum-freshess.

The way it happened to Cargo itself is obvious:
When running a cargo built from source with old build cache not aware of the new -Zchecksum-freshness fingerprint dep-info format,
Cargo itself couldn't distinguish the difference.
Instead it tried to read dep-info from the same location,
and then got an OOM because the format is not compatible.

For the PackSquash repo, it is a bit subtle. The reproduction uses two different nightly versions. Cargo passes -C metadata (the hash appended after <pkg>-<ver> coordinate) to distinguish different output artifact for different set of flags and features. However, if you're using nightly, Cargo doesn't really separate different versions of nightly. It ends up putting in the same directory. This is somehow intentional however depending on fingerprint to correctly detect a rebuild is needed. Before fingerprint ever got started, Cargo loaded the fingerprint dep-info files and boo!

BTW, the first bad dep-info files in PackSquash is target/debug/.fingerprint/packsquash_cli-5a0a48e3ab0b2ec9/dep-build-script-build-script-build, which in hex view looks like