supplied instant is later than self #86470
Labels
A-time
Area: Time
C-bug
Category: This is a bug.
I-ICE
Issue: The compiler panicked, giving an Internal Compilation Error (ICE) ❄️
T-compiler
Relevant to the compiler team, which will review and decide on the PR/issue.
Comments
ghost
added
C-bug
|
Can you check your system clock? That might be the issue. Also did you use |
|
This usually is due to outdated kernels, hardware or hypervisor bugs (see #80674, #82606, #84448 and several others) Can you provide the following information?
|
|
@the8472 We encountered the same issue in a production service (TiKV) deployed on AWS EC2. System information as requested:
|
|
@tabokie thanks for the report.
Looks like that does contain backported patches which query the hypervisor whether the pvclock is stable which then disables the clock monotonization in the kernel So either the version of xen that amazon is using has a bug (advertising a stable clock when it isn't) or the backported patches are missing some critical pieces. How reliably can you reproduce the issue? If it occurs frequently enough to make investigation easy, can you update your kernel to see if that solves it? That would indicate some incomplete backports. |
|
@the8472 Thanks for looking into our case, we have only experienced the panic once. Will share more information if there's progress in reproducing it. Since it's hard for us to lock-in certain "safe" kernel/hardware to our software, or check the safety of arbitrary kernel/hardware, we are leaning towards replacing standard |
|
I'm primarily asking to figure out how to adjust std's built-in heuristics for the clock monotonization and where to report upstream bugs. Something is violating the posix CLOCK_MONOTONIC contract and it's hard to pinpoint the cause because the issue only affects specific hardware/software constellations.
You can already get this with existing standard library methods via |
|
For me, this error did not occur in a VM, it was on bare metal. Unfortunately said machine isn't working at the moment, but I do know the CPU was a Celeron n3060. |
|
I'm seeing this in a macOS Catalina (10.15.6) VM, running in virtualbox (6.1.26) on a macOS Big Sur (11.5.1) host. Guest VM: rustc 1.53.0 |
|
@frankosterfeld assuming the issue doesn't occur on the host system it sounds like a bug in the way that VirtualBox provides timers to the guest. Anyway, can you provide a backtrace? |
|
@the8472 Indeed, it doesn't seem to happen on the host (I built 5 times now without issue). Googling I find (old) issues like this one: https://www.virtualbox.org/ticket/7915 Here's a backtrace: Backtracethread 'rustc' panicked at 'supplied instant is later than self', library/std/src/time.rs:281:48
stack backtrace:
0: 0x100afa88e - ::fmt::h7dac4b58aa11870a
1: 0x100b9484e - core::fmt::write::h53badd02f711efbe
thread 'main' panicked at 'supplied instant is later than self', library/std/src/time.rs:281:48er, memoffset(build.rs)
stack backtrace:
0: 0x103c6a5ce - ::fmt::h7dac4b58aa11870a
1: 0x103caaaee - core::fmt::write::h53badd02f711efbe
2: 0x103c65c7a - std::io::Write::write_fmt::hd02c4bbaf6463567
3: 0x103c853df - std::panicking::default_hook::{{closure}}::ha7d22468d4529f02
4: 0x103c84f6a - std::panicking::default_hook::hc6c5fb91e7ed228a
5: 0x103c85950 - std::panicking::rust_panic_with_hook::h94fd567d38cd4da7
6: 0x103c6a945 - std::panicking::begin_panic_handler::{{closure}}::h953a8f32e4882703
7: 0x103c6a748 - std::sys_common::backtrace::__rust_end_short_backtrace::h8472811b9ff0c597
8: 0x103c854e3 - _rust_begin_unwind
9: 0x103d04bff - core::panicking::panic_fmt::h5a271c2839c7a480
10: 0x103d04e9a - core::option::expect_failed::h7ae81fcabe446ed2
11: 0x103c66772 - std::sys::unix::condvar::Condvar::wait_timeout::h0a053b632c3af75a
12: 0x103832472 - std::sync::condvar::Condvar::wait_timeout_while::h64f3c7014de50a03
13: 0x103628a55 - cargo::util::queue::Queue::pop::h62751c7b271dfc21
14: 0x1036c43fb - cargo::core::compiler::job_queue::DrainState::drain_the_queue::h2cdc48f1a4c2b8d3
15: 0x1037a75fb - std::panic::catch_unwind::h7ce81730fa67d5fb
16: 0x103736dbf - crossbeam_utils::thread::scope::h5dd3bef14adde2e1
17: 0x1036c233f - cargo::core::compiler::job_queue::JobQueue::execute::h7220b2748065b72a
18: 0x10364546a - cargo::core::compiler::context::Context::compile::hd88a984ecb38f964
19: 0x1038eaffb - cargo::ops::cargo_compile::compile_ws::hdfefe68826251154
20: 0x1036d4083 - cargo::ops::cargo_install::install_one::hd815a53823f4cf4c
21: 0x1036cea5b - cargo::ops::cargo_install::install::h2a5d5b9693d18de7
22: 0x10352c6c7 - cargo::commands::install::exec::had4b593b99923474
23: 0x1035108da - cargo::cli::main::h6d1be6cd0c776121
24: 0x1035220ad - cargo::main::h7fb76a874797df25
25: 0x10350bc3a - std::sys_common::backtrace::__rust_begin_short_backtrace::hcee2bb4cb8c63cbc
26: 0x10350be1c - std::rt::lang_start::{{closure}}::haa89dd94abd14a82
27: 0x103c8f6da - std::rt::lang_start_internal::h43b407d2a4fa0408
28: 0x1035243c9 - _main
thread '' panicked at 'called `Result::unwrap()` on an `Err` value: PoisonError { .. }', src/cargo/util/queue.rs:43:46
stack backtrace:
0: 0x103c6a5ce - ::fmt::h7dac4b58aa11870a
1: 0x103caaaee - core::fmt::write::h53badd02f711efbe
2: 0x103c65c7a - std::io::Write::write_fmt::hd02c4bbaf6463567
3: 0x103c853df - std::panicking::default_hook::{{closure}}::ha7d22468d4529f02
4: 0x103c84f6a - std::panicking::default_hook::hc6c5fb91e7ed228a
5: 0x103c85950 - std::panicking::rust_panic_with_hook::h94fd567d38cd4da7
6: 0x103c6a945 - std::panicking::begin_panic_handler::{{closure}}::h953a8f32e4882703
7: 0x103c6a748 - std::sys_common::backtrace::__rust_end_short_backtrace::h8472811b9ff0c597
8: 0x103c854e3 - _rust_begin_unwind
9: 0x103d04bff - core::panicking::panic_fmt::h5a271c2839c7a480
10: 0x103d04f15 - core::result::unwrap_failed::hdc9b54af28d7ad5c
11: 0x10362893a - cargo::util::queue::Queue::push_bounded::h7b56b7ce7acce761
12: 0x1036c0470 - cargo::core::compiler::job_queue::JobState::stderr::h382d9befd36f7229
13: 0x10392726e - cargo::core::compiler::on_stderr_line::h5a0db9e2e6b9f88e
14: 0x1039c988a - cargo::core::compiler::rustc::{{closure}}::{{closure}}::had17844247424009
15: 0x103c418c1 - cargo_util::process_builder::ProcessBuilder::exec_with_streaming::{{closure}}::{{closure}}::hd6767f2b856efd8d
16: 0x103c3960b - cargo_util::read2::imp::read2::ha62124b9033095ba
17: 0x103c40f14 - cargo_util::process_builder::ProcessBuilder::exec_with_streaming::h60ef7d080ecc11a6
18: 0x10391c78b - ::exec::h509c01b15eb5a353
19: 0x1039a836e - core::ops::function::FnOnce::call_once{{vtable.shim}}::hc603d9f174e99086
20: 0x1039a5881 - core::ops::function::FnOnce::call_once{{vtable.shim}}::ha4f513e6806dc991
21: 0x1039a5881 - core::ops::function::FnOnce::call_once{{vtable.shim}}::ha4f513e6806dc991
22: 0x103739532 - core::ops::function::FnOnce::call_once{{vtable.shim}}::hc5c77f1c61817351
23: 0x1036a43e3 - std::sys_common::backtrace::__rust_begin_short_backtrace::h47796c4d5f4f430e
24: 0x1036ee18d - core::ops::function::FnOnce::call_once{{vtable.shim}}::hf2d6576490aa894e
25: 0x103c66a9b - std::sys::unix::thread::Thread::new::thread_start::h26c063646f1d014f
26: 0x7fff6ff40109 - __pthread_start
thread '' panicked at 'called `Result::unwrap()` on an `Err` value: PoisonError { .. }', src/cargo/util/queue.rs:37:27
stack backtrace:
0: 0x103c6a5ce - ::fmt::h7dac4b58aa11870a
1: 0x103caaaee - core::fmt::write::h53badd02f711efbe
2: 0x103c65c7a - std::io::Write::write_fmt::hd02c4bbaf6463567
3: 0x103c853df - std::panicking::default_hook::{{closure}}::ha7d22468d4529f02
4: 0x103c84f6a - std::panicking::default_hook::hc6c5fb91e7ed228a
5: 0x103c85950 - std::panicking::rust_panic_with_hook::h94fd567d38cd4da7
6: 0x103c6a945 - std::panicking::begin_panic_handler::{{closure}}::h953a8f32e4882703
7: 0x103c6a748 - std::sys_common::backtrace::__rust_end_short_backtrace::h8472811b9ff0c597
8: 0x103c854e3 - _rust_begin_unwind
9: 0x103d04bff - core::panicking::panic_fmt::h5a271c2839c7a480
10: 0x103d04f15 - core::result::unwrap_failed::hdc9b54af28d7ad5c
11: 0x103628734 - cargo::util::queue::Queue::push::hf4c76d68a8b34f4f
12: 0x10373b09d - core::ptr::drop_in_place::hf8b6a511468ec05f
13: 0x103739800 - core::ops::function::FnOnce::call_once{{vtable.shim}}::hc5c77f1c61817351
14: 0x1036a43e3 - std::sys_common::backtrace::__rust_begin_short_backtrace::h47796c4d5f4f430e
15: 0x1036ee18d - core::ops::function::FnOnce::call_once{{vtable.shim}}::hf2d6576490aa894e
16: 0x103c66a9b - std::sys::unix::thread::Thread::new::thread_start::h26c063646f1d014f
17: 0x7fff6ff40109 - __pthread_start
thread panicked while panicking. aborting.
And another one: Backtrace 2thread 'rustc' panicked at 'supplied instant is later than self', library/std/src/time.rs:281:48
stack backtrace:
0: 0x1007a988e - ::fmt::h7dac4b58aa11870a
1: 0x10084384e - core::fmt::write::h53badd02f711efbe
2: 0x1007a3aca - std::io::Write::write_fmt::hd02c4bbaf6463567
3: 0x1007cbebf - std::panicking::default_hook::{{closure}}::ha7d22468d4529f02
4: 0x1007cba4a - std::panicking::default_hook::hc6c5fb91e7ed228a
5: 0x103135748 - rustc_driver::report_ice::hddaa632f5d60e0de
6: 0x1007cc452 - std::panicking::rust_panic_with_hook::h94fd567d38cd4da7
7: 0x1007aa485 - std::panicking::begin_panic_handler::{{closure}}::h953a8f32e4882703
8: 0x1007a9a08 - std::sys_common::backtrace::__rust_end_short_backtrace::h8472811b9ff0c597
9: 0x1007cbfc3 - _rust_begin_unwind
10: 0x1008568bf - core::panicking::panic_fmt::h5a271c2839c7a480
11: 0x100856b7a - core::option::expect_failed::h7ae81fcabe446ed2
12: 0x10079b585 - std::time::Instant::elapsed::h66eaa7e5ce5cd27b
13: 0x1033ddd76 - rustc_codegen_llvm::base::compile_codegen_unit::hf7f6012ae003f573
14: 0x103376e34 - rustc_codegen_ssa::base::codegen_crate::h4a39f71ab2586f47
15: 0x1033a6673 - ::codegen_crate::h049265f985b50098
16: 0x1032516df - rustc_interface::passes::QueryContext::enter::h397e9c5bcf7ba8fc
17: 0x103272ce0 - rustc_interface::queries::Queries::ongoing_codegen::h408a077800f099c5
18: 0x10313f496 - rustc_interface::queries::::enter::h58bca51b0e877876
19: 0x1031378d1 - rustc_span::with_source_map::hbc85b87c1f294b1d
20: 0x10314018f - rustc_interface::interface::create_compiler_and_run::h0c3a96672037be70
21: 0x103162506 - scoped_tls::ScopedKey::set::hf07f944fc712341d
22: 0x103164619 - std::sys_common::backtrace::__rust_begin_short_backtrace::hacfc6003a415e17f
23: 0x10316ea2d - core::ops::function::FnOnce::call_once{{vtable.shim}}::hd8bfde75fe9c3c41
24: 0x1007a5aab - std::sys::unix::thread::Thread::new::thread_start::h26c063646f1d014f
25: 0x7fff6ff40109 - __pthread_start
|
olix0r
added a commit
to olix0r/hyper
that referenced
this issue
Jan 31, 2022
We have reports of runtime panics (linkerd/linkerd2#7748) that sound a lot like rust-lang/rust#86470. We don't have any evidence that these panics originate in hyper, but hyperium#2385 reports a similar issue. Even though this is almost definitely a bug in Rust, it seems most prudent to actively avoid the uses of `Instant` that are prone to this bug. This change replaces uses of `Instant::elapsed` and `Instant::sub` with calls to `Instant::saturating_duration_since` to prevent this class of panic.
olix0r
added a commit
to tower-rs/tower
that referenced
this issue
Jan 31, 2022
We have reports of runtime panics (linkerd/linkerd2#7748) that sound a lot like rust-lang/rust#86470. We don't have any evidence that these panics originate in hyper, but #2385 reports a similar issue. Even though this is almost definitely a bug in Rust, it seems most prudent to actively avoid the uses of `Instant` that are prone to this bug. This change replaces uses of `Instant::elapsed` and `Instant::sub` with calls to `Instant::saturating_duration_since` to prevent this class of panic. These fixes should ultimately be made in the standard library, but this change lets us avoid this problem while we wait for those fixes. See also hyperium/hyper#2746
olix0r
added a commit
to tower-rs/tower
that referenced
this issue
Jan 31, 2022
We have reports of runtime panics (linkerd/linkerd2#7748) that sound a lot like rust-lang/rust#86470. We don't have any evidence that these panics originate in tower, but we have some potentialy flawed `Instant` arithmetic that could panic in this way. Even though this is almost definitely a bug in Rust, it seems most prudent to actively avoid the uses of `Instant` that are prone to this bug. This change replaces uses of `Instant::elapsed` and `Instant::sub` with calls to `Instant::saturating_duration_since` to prevent this class of panic. These fixes should ultimately be made in the standard library, but this change lets us avoid this problem while we wait for those fixes. See also hyperium/hyper#2746
olix0r
added a commit
to tower-rs/tower
that referenced
this issue
Jan 31, 2022
We have reports of runtime panics (linkerd/linkerd2#7748) that sound a lot like rust-lang/rust#86470. We don't have any evidence that these panics originate in tower, but we have some potentialy flawed `Instant` arithmetic that could panic in this way. Even though this is almost definitely a bug in Rust, it seems most prudent to actively avoid the uses of `Instant` that are prone to this bug. This change replaces uses of `Instant::elapsed` and `Instant::sub` with calls to `Instant::saturating_duration_since` to prevent this class of panic. These fixes should ultimately be made in the standard library, but this change lets us avoid this problem while we wait for those fixes. See also hyperium/hyper#2746
olix0r
added a commit
to hyperium/h2
that referenced
this issue
Jan 31, 2022
We have reports of runtime panics (linkerd/linkerd2#7748) that sound a lot like rust-lang/rust#86470. We don't have any evidence that these panics originate in h2, but there is one use of `Instant::sub` that could panic in this way. Even though this is almost definitely a bug in Rust, it seems most prudent to actively avoid the uses of `Instant` that are prone to this bug. These fixes should ultimately be made in the standard library, but this change lets us avoid this problem while we wait for those fixes. This change replaces uses of `Instant::elapsed` and `Instant::sub` with calls to `Instant::saturating_duration_since` to prevent this class of panic. See also hyperium/hyper#2746
seanmonstar
pushed a commit
to hyperium/h2
that referenced
this issue
Feb 1, 2022
We have reports of runtime panics (linkerd/linkerd2#7748) that sound a lot like rust-lang/rust#86470. We don't have any evidence that these panics originate in h2, but there is one use of `Instant::sub` that could panic in this way. Even though this is almost definitely a bug in Rust, it seems most prudent to actively avoid the uses of `Instant` that are prone to this bug. These fixes should ultimately be made in the standard library, but this change lets us avoid this problem while we wait for those fixes. This change replaces uses of `Instant::elapsed` and `Instant::sub` with calls to `Instant::saturating_duration_since` to prevent this class of panic. See also hyperium/hyper#2746
seanmonstar
pushed a commit
to tower-rs/tower
that referenced
this issue
Feb 1, 2022
We have reports of runtime panics (linkerd/linkerd2#7748) that sound a lot like rust-lang/rust#86470. We don't have any evidence that these panics originate in tower, but we have some potentialy flawed `Instant` arithmetic that could panic in this way. Even though this is almost definitely a bug in Rust, it seems most prudent to actively avoid the uses of `Instant` that are prone to this bug. This change replaces uses of `Instant::elapsed` and `Instant::sub` with calls to `Instant::saturating_duration_since` to prevent this class of panic. These fixes should ultimately be made in the standard library, but this change lets us avoid this problem while we wait for those fixes. See also hyperium/hyper#2746
olix0r
added a commit
to tokio-rs/tokio
that referenced
this issue
Feb 1, 2022
`Instant::duration_since`, `Instant::elapsed`, and `Instant::sub` may panic. This is especially dangerous when `Instant::now` travels back in time. While this isn't supposed to happen, this behavior is highly platform-dependent (e.g., rust-lang/rust#86470). This change modifies the behavior of `tokio::time::Instant` to prevent this class of panic, as proposed for `std::time::Instant` in rust-lang/rust#89926.
Darksonn
pushed a commit
to tokio-rs/tokio
that referenced
this issue
Feb 6, 2022
`Instant::duration_since`, `Instant::elapsed`, and `Instant::sub` may panic. This is especially dangerous when `Instant::now` travels back in time. While this isn't supposed to happen, this behavior is highly platform-dependent (e.g., rust-lang/rust#86470). This change modifies the behavior of `tokio::time::Instant` to prevent this class of panic, as proposed for `std::time::Instant` in rust-lang/rust#89926.
ellenhp
pushed a commit
to killyourphone/tokio
that referenced
this issue
Feb 6, 2022
`Instant::duration_since`, `Instant::elapsed`, and `Instant::sub` may panic. This is especially dangerous when `Instant::now` travels back in time. While this isn't supposed to happen, this behavior is highly platform-dependent (e.g., rust-lang/rust#86470). This change modifies the behavior of `tokio::time::Instant` to prevent this class of panic, as proposed for `std::time::Instant` in rust-lang/rust#89926.
|
|
I can consistently reproduce it on a machine running 24/7 for the past month. rustc --version --verbose
cat /etc/os-release
cat /proc/cpuinfo
cat /sys/devices/system/clocksource/clocksource0/current_clocksource
uname -rv
See the stacktrace for the |
|
#89926 landed recently, nightlies and 1.60 onwards will no longer panic in this situation. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Meta
rustc --version --verbose:Error output
Backtrace
This might be an LLVM bug, as it never occurs on Windows and occurs in every rust program that I've compiled on Linux, not when compiling, but running it.
The text was updated successfully, but these errors were encountered: