found unstable fingerprints for extern_mod_stmt_cnum

[est31]

I encountered this bug in the CI for my rcgen PR: rustls/rcgen#53

full log link

Running `rustc --crate-name botan tests/botan.rs --error-format=json --json=diagnostic-rendered-ansi --emit=dep-info,link -C embed-bitcode=no -C debuginfo=2 --test --cfg 'feature="default"' --cfg 'feature="pem"' --cfg 'feature="x509-parser"' -C metadata=96712806c8dc92bf -C extra-filename=-96712806c8dc92bf --out-dir /Users/runner/work/rcgen/rcgen/target/debug/deps -C incremental=/Users/runner/work/rcgen/rcgen/target/debug/incremental -L dependency=/Users/runner/work/rcgen/rcgen/target/debug/deps --extern botan=/Users/runner/work/rcgen/rcgen/target/debug/deps/libbotan-0a4d2a3a26fb1b22.rlib --extern chrono=/Users/runner/work/rcgen/rcgen/target/debug/deps/libchrono-2edffe8c4cbe14a9.rlib --extern openssl=/Users/runner/work/rcgen/rcgen/target/debug/deps/libopenssl-921daadd081c933a.rlib --extern pem=/Users/runner/work/rcgen/rcgen/target/debug/deps/libpem-f00ebd26ddff9dd7.rlib --extern rcgen=/Users/runner/work/rcgen/rcgen/target/debug/deps/librcgen-bd616741af37ab36.rlib --extern ring=/Users/runner/work/rcgen/rcgen/target/debug/deps/libring-70ea1056c377f123.rlib --extern webpki=/Users/runner/work/rcgen/rcgen/target/debug/deps/libwebpki-9c88c2488ba93ec8.rlib --extern x509_parser=/Users/runner/work/rcgen/rcgen/target/debug/deps/libx509_parser-46230be220d16bf1.rlib --extern yasna=/Users/runner/work/rcgen/rcgen/target/debug/deps/libyasna-6f4481586b48639f.rlib -D warnings -L native=/Users/runner/work/rcgen/rcgen/target/debug/build/botan-sys-0bdd76b27e0e28d0/out/botan -L 'native=/usr/local/opt/openssl@1.1/lib' -L native=/Users/runner/work/rcgen/rcgen/target/debug/build/ring-7c4a53e107b77dd5/out`
thread 'rustc' panicked at 'found unstable fingerprints for extern_mod_stmt_cnum(rcgen[846b]::rcgen)', /rustc/acca818928654807ed3bc1ce0e97df118f8716c8/compiler/rustc_query_system/src/query/plumbing.rs:593:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: rustc 1.52.0-nightly (acca81892 2021-03-13) running on x86_64-apple-darwin

note: compiler flags: -C embed-bitcode=no -C debuginfo=2 -C incremental --crate-type bin

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
#0 [extern_mod_stmt_cnum] computing crate imported by `rcgen`
#1 [check_mod_unstable_api_usage] checking for unstable API usage in top-level module
end of query stack
error: could not compile `rcgen`

Caused by:
  process didn't exit successfully: `rustc --crate-name rcgen src/main.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debuginfo=2 --cfg 'feature="default"' --cfg 'feature="pem"' --cfg 'feature="x509-parser"' -C metadata=e67008cd305628cc --out-dir /Users/runner/work/rcgen/rcgen/target/debug/deps -C incremental=/Users/runner/work/rcgen/rcgen/target/debug/incremental -L dependency=/Users/runner/work/rcgen/rcgen/target/debug/deps --extern chrono=/Users/runner/work/rcgen/rcgen/target/debug/deps/libchrono-2edffe8c4cbe14a9.rlib --extern pem=/Users/runner/work/rcgen/rcgen/target/debug/deps/libpem-f00ebd26ddff9dd7.rlib --extern rcgen=/Users/runner/work/rcgen/rcgen/target/debug/deps/librcgen-bd616741af37ab36.rlib --extern ring=/Users/runner/work/rcgen/rcgen/target/debug/deps/libring-70ea1056c377f123.rlib --extern x509_parser=/Users/runner/work/rcgen/rcgen/target/debug/deps/libx509_parser-46230be220d16bf1.rlib --extern yasna=/Users/runner/work/rcgen/rcgen/target/debug/deps/libyasna-6f4481586b48639f.rlib -D warnings -L native=/Users/runner/work/rcgen/rcgen/target/debug/build/ring-7c4a53e107b77dd5/out` (exit code: 101)
warning: build failed, waiting for other jobs to finish...

[Mark-Simulacrum]

cc @Aaron1011

[est31]

Backtrace:

 thread 'rustc' panicked at 'found unstable fingerprints for extern_mod_stmt_cnum(rcgen[846b]::rcgen)', /rustc/acca818928654807ed3bc1ce0e97df118f8716c8/compiler/rustc_query_system/src/query/plumbing.rs:593:5
stack backtrace:
   0: _rust_begin_unwind
   1: std::panicking::begin_panic_fmt
   2: rustc_query_system::query::plumbing::incremental_verify_ich
   3: rustc_query_system::query::plumbing::load_from_disk_and_cache_in_memory
   4: rustc_data_structures::stack::ensure_sufficient_stack
   5: rustc_query_system::query::plumbing::get_query_impl
   6: <rustc_query_impl::Queries as rustc_middle::ty::query::QueryEngine>::extern_mod_stmt_cnum
   7: <rustc_passes::stability::Checker as rustc_hir::intravisit::Visitor>::visit_item
   8: rustc_middle::hir::map::Map::visit_item_likes_in_module
   9: rustc_passes::stability::check_mod_unstable_api_usage
  10: rustc_middle::dep_graph::<impl rustc_query_system::dep_graph::DepKind for rustc_middle::dep_graph::dep_node::DepKind>::with_deps
  11: rustc_query_system::dep_graph::graph::DepGraph<K>::with_task_impl
  12: rustc_data_structures::stack::ensure_sufficient_stack
  13: rustc_query_system::query::plumbing::force_query_with_job
  14: rustc_query_system::query::plumbing::get_query_impl
  15: <rustc_query_impl::Queries as rustc_middle::ty::query::QueryEngine>::check_mod_unstable_api_usage
  16: std::panic::catch_unwind
  17: rustc_session::utils::<impl rustc_session::session::Session>::time
  18: rustc_interface::passes::analysis
  19: rustc_middle::dep_graph::<impl rustc_query_system::dep_graph::DepKind for rustc_middle::dep_graph::dep_node::DepKind>::with_deps
  20: rustc_query_system::dep_graph::graph::DepGraph<K>::with_task_impl
  21: rustc_data_structures::stack::ensure_sufficient_stack
  22: rustc_query_system::query::plumbing::force_query_with_job
  23: rustc_query_system::query::plumbing::get_query_impl
  24: <rustc_query_impl::Queries as rustc_middle::ty::query::QueryEngine>::analysis
  25: rustc_interface::passes::QueryContext::enter
  26: rustc_interface::queries::<impl rustc_interface::interface::Compiler>::enter
  27: rustc_span::with_source_map
  28: scoped_tls::ScopedKey<T>::set
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

[Aaron1011]

@est31: Have you been able to reproduce this on Linux?

[Aaron1011]

I think the root cause is with how extern_mod_stmt_cnum is defined:

rust/compiler/rustc_middle/src/query/mod.rs

Lines 1280 to 1282 in bb4cdf8

	query extern_mod_stmt_cnum(def_id: LocalDefId) -> Option<CrateNum> {
	desc { |tcx| "computing crate imported by `{}`", tcx.def_path_str(def_id.to_def_id()) }
	}

This query takes in the LocalDefId of an extern crate statement, and returns the CrateNum of the crate being imported. However, this does not fit the query model - the input is a local definition (which implicitly depends on the local crate's name and disambiguator), and returns a foreign definition. The output is not actually a pure function of the input - it's possible to have the local crate be entierly unchanged, but have the foreign crate end up with a different StableCrateId (due to getting a different disambiguator). This is reflected in the fact that the query implementation uses the global tcx.extern_crate_map, which does not go through the query system.

[Aaron1011]

I think the solution is to mark the query as eval_always

[est31]

Hmmm I'm getting it on the master branch too not just that PR. So it seems to be a regression that happened roughly between rustc from 18 days ago and rustc 1.52.0-nightly (acca81892 2021-03-13)

[Aaron1011]

@est31 This is expected breakage from PR #83007 - we're now catching bugs in queries that were previously ignored.

[Mark-Simulacrum]

It looks like the query does a pretty simple hash lookup - is there a reason for it to be a query? If it only takes LocalDefId it's presumably not used cross-crate at all, so it seems like it should just be a function on tcx?

[Aaron1011]

I've opened #83128, but this issue still needs a regression test.

[est31]

Ok so I've done some testing on the crush_ice_83126 branch of rcgen and it seems the following triggers the bug:

crate a: empty lib.rs (I just used byteorder but I guess it should also work with empty lib.rs)

crate b Cargo.toml:

[dependencies]
a = { path=".....", optional = true }

crate b lib.rs, tests/foo.rs: empty

crate b main.rs:

extern crate b; // This seems important!
fn main() {}

Then you invoke cargo twice:

cargo test
cargo test --features a

Have you been able to reproduce this on Linux?

No reproduction on Ubuntu but on macOS and Windows the CI can reproduce it just fine. Latest nightly or nightly-2021-03-14 from rustup do the job.

[est31]

A reproduction on Ubuntu has been found (not minimized yet though): EmbarkStudios/rust-gpu#493 (comment)

[est31]

New PR to tackle the bug: #83153

[est31]

@rustbot label E-needs-test