ICE: "cannot lex source_file without source" with integration test

[XAMPPRocky]

If you attempt to include! a file in both the library and an external integration test that links to the library (this is important because if there is no extern crate or use statements, the ICE does not happen), it causes an ICE in the latest nightly. This ICE is not present in stable or beta.

build.rs

fn main() {
    std::fs::write(
        &std::path::Path::new(&std::env::var_os("OUT_DIR").unwrap()).join("crash.rs"),
        "pub struct A;",
    )
    .unwrap();
}

src/lib.rs

include!(concat!(env!("OUT_DIR"), "/", "crash.rs"));

tests/crash.rs

extern crate abc; // abc must be replaced by the name of the crate to test (e.g. the name in the Cargo.toml file)

include!(concat!(env!("OUT_DIR"), "/", "crash.rs"));

Command

cargo test

Meta

rustc --version --verbose:

rustc 1.54.0-nightly (625d5a693 2021-06-01)
binary: rustc
commit-hash: 625d5a693e4697bcafdd34fd1a38c281acabb8e9
commit-date: 2021-06-01
host: x86_64-apple-darwin
release: 1.54.0-nightly
LLVM version: 12.0.1

Error output

error: internal compiler error: cannot lex `source_file` without source: /Users/src/ice-test/target/debug/build/ice-test-dcdce532143b8bc1/out/xds.core.v3.rs

thread 'rustc' panicked at 'Box<Any>', compiler/rustc_errors/src/lib.rs:1005:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: rustc 1.54.0-nightly (625d5a693 2021-06-01) running on x86_64-apple-darwin

note: compiler flags: -C embed-bitcode=no -C split-debuginfo=unpacked -C debuginfo=2 -C incremental

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
end of query stack

Backtrace

stack backtrace:
   0:        0x1170d13b4 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hc0a32d3c3444fdc6
   1:        0x117135b5c - core::fmt::write::h120e3853e954a389
   2:        0x1170c33ea - std::io::Write::write_fmt::hfa38fa23f7fce513
   3:        0x1170d506f - std::panicking::default_hook::{{closure}}::h5d5d1571ac2df726
   4:        0x1170d4b6d - std::panicking::default_hook::h4f38c2182113a3d6
   5:        0x10f8b72b8 - rustc_driver::report_ice::h315ec60d4f33b035
   6:        0x1170d58a6 - std::panicking::rust_panic_with_hook::h111fa41f5cc5bae1
   7:        0x1139d22b1 - std::panicking::begin_panic::{{closure}}::hb5d77af7cdcca810
   8:        0x1139d2229 - std::sys_common::backtrace::__rust_end_short_backtrace::h4cf6426c0e11596d
   9:        0x113c91811 - std::panicking::begin_panic::hf129750fee14a661
  10:        0x1139f1080 - std::panic::panic_any::h35ee1f3d736053e5
  11:        0x1139f76ac - rustc_errors::HandlerInner::bug::hffe2549ce25674c9
  12:        0x1139f5457 - rustc_errors::Handler::bug::h5d19ee9d6012acf4
  13:        0x11335dc2a - rustc_parse::maybe_file_to_stream::{{closure}}::h2b76850df2689f69
  14:        0x11335db4f - rustc_parse::maybe_file_to_stream::h6a5b54720a5326c4
  15:        0x11335d5e3 - rustc_parse::maybe_source_file_to_parser::hb2889c8806f22353
  16:        0x11335d244 - rustc_parse::new_parser_from_file::h880749fe6c165a55
  17:        0x111edea8e - rustc_builtin_macros::source_util::expand_include::hec829b007a51890d
  18:        0x1131665fa - rustc_expand::expand::MacroExpander::fully_expand_fragment::hc9471323bbd298aa
  19:        0x113165489 - rustc_expand::expand::MacroExpander::expand_crate::ha260ef6cedc5f864
  20:        0x10f9c91c0 - rustc_interface::passes::configure_and_expand_inner::h995c2829b63caf5c
  21:        0x10f9bf430 - rustc_interface::passes::configure_and_expand::{{closure}}::hc06a0520c8089ddc
  22:        0x10f9b2fdb - rustc_data_structures::box_region::PinnedGenerator<I,A,R>::new::h7057a1aefc5c0c79
  23:        0x10f9c7ef5 - rustc_interface::passes::configure_and_expand::h5530d33ac5f81757
  24:        0x10f9e680d - rustc_interface::queries::Queries::expansion::hde0b694af7141ffa
  25:        0x10f8d5e89 - rustc_span::with_session_globals::h8ceb270caa69a3cc
  26:        0x10f913b3f - std::sys_common::backtrace::__rust_begin_short_backtrace::h044fc2ce181347b9
  27:        0x10f8c6ebd - core::ops::function::FnOnce::call_once{{vtable.shim}}::h95785b84dcf6930e
  28:        0x1170e0b6b - std::sys::unix::thread::Thread::new::thread_start::hd7d90a4d6d405897
  29:     0x7fff2073f954 - __pthread_start

[apiraino]

@XAMPPRocky I've tried reproducing on linux:

$ rustup default nightly-2021-XX-YY-x86_64-unknown-linux-gnu
$ git clone --depth=1 https://github.com/XAMPPRocky/tonic-ice
$ cd tonic-ice && cargo build

where XX and YY are recent nightlies (625d5a693 2021-06-01 included), but on my machine it never fails. Can you help me reproduce the ICE? Maybe I'm missing some compilation flags?

[XAMPPRocky]

@apiraino D'oh, I forgot to include the command. You need to run cargo test.

[apiraino]

ok I think nailed it to megacommit e1ff91f from PR #83813

searched nightlies: from nightly-2021-05-01 to nightly-2021-06-02
regressed nightly: nightly-2021-05-13
searched commits: from 5c02926 to 21e92b9
regressed commit: e1ff91f

bisected with cargo-bisect-rustc v0.6.0

Host triple: x86_64-unknown-linux-gnu
Reproduce with:

$ cat test.sh
#!/bin/sh
cargo test

$ cargo bisect-rustc --start=2021-05-01 --script=./test.sh --regress ice

[hellow554]

Here's a smaller example: Three files are needed as for now:

build.rs

fn main() {
    std::fs::write(
        &std::path::Path::new(&std::env::var_os("OUT_DIR").unwrap()).join("crash.rs"),
        "pub struct A;",
    )
    .unwrap();
}

src/lib.rs

include!(concat!(env!("OUT_DIR"), "/", "crash.rs"));

tests/crash.rs

extern crate abc; // abc must be replaced by the name of the crate to test (e.g. the name in the Cargo.toml file)

include!(concat!(env!("OUT_DIR"), "/", "crash.rs"));

afterwards cargo test will result in the ICE.

Here's a small script that does it for you:

Bash script for settings up the testcase

#!/bin/bash

cat << BLOCK > Cargo.toml
[package]
name = "abc"
version = "0.1.0"
edition = "2018"
BLOCK

mkdir -p src tests

cat << BLOCK > build.rs
fn main() {
    std::fs::write(
        &std::path::Path::new(&std::env::var_os("OUT_DIR").unwrap()).join("crash.rs"),
        "pub struct A;",
    )
    .unwrap();
}
BLOCK

cat << BLOCK > src/lib.rs
include!(concat!(env!("OUT_DIR"), "/", "crash.rs"));
BLOCK

cat << BLOCK > tests/crash.rs
extern crate abc;
include!(concat!(env!("OUT_DIR"), "/", "crash.rs"));
BLOCK

cargo test

[XAMPPRocky]

@hellow554 Thank you for reducing it! I've moved that to the top of the issue.

[apiraino]

Assigning priority as discussed in the Zulip thread of the Prioritization Working Group.

@rustbot label -I-prioritize +P-medium

(cc @cbeuw as PR #83813 author, might have insights here)

[michaelwoerister]

The underlying problem is that the same source file is first imported from the upstream crate (with src == None) and then also allocated in the local crate where the source code is available (and required). Because the source map already has an entry for the file from the first step it re-uses that one.

I'm not sure yet what the best solution for this is. Updating the SourceFile in-place in the second step seems problematic. Maybe importing should try to load the source for local files.

[gilescope]

Hit by this too.