Backtraces don't work during ICE #71060
Labels
C-bug
Category: This is a bug.
O-windows-gnu
Toolchain: GNU, Operating system: Windows
T-compiler
Relevant to the compiler team, which will review and decide on the PR/issue.
Comments
This comment has been minimized.
This comment has been minimized.
jonas-schievink
added
the
T-compiler
Merged
|
My guess is that this is highly likely to be caused by rust-lang/backtrace-rs#333. Dynamic libraries are not found by backtrace-rs at this time and symbolicated at all. Currently I don't know how to iterate over loaded libraries and register what address ranges they occupy. |
|
@spastorino this might be something for wg-prioritization to talk about. |
|
@mati865 do you want wg-prioritization to prioritize this issue? or do you want to nominate it for discussion on a |
|
@spastorino I just want "increase visibility" of this issue hoping that somebody who knows about this stuff would step in. Sorry if that was "wrong address". |
|
@mati865 not at all, it's not the wrong address, I'm trying to understand how can we help. If you consider this issue important and want to increase its visibility, you can nominate the issue for discussion on the next |
|
@spastorino thank you for the explanation, although I don't think I can use @rustbot modify labels: +I-nominated Reason for nomination: |
|
Error: Label I-nominated can only be set by Rust team members Please let |
|
@rustbot ping windows |
|
Error: This team ( Please let |
|
Ohh right, there are still some pending PRs to be able to ping windows notify group. Doing so manually for now ... cc @arlosi @danielframpton @gdr-at-ms @kennykerr @luqmana @lzybkr @retep998 @rylev @sivadeilra |
|
@manodasanW may be able to help. He is our resident debugging expert, but doesn't necessarily know anything about Rust. 🙂 |
|
@alexcrichton - Regarding checking which libraries are loaded - you can enumerate the memory regions and query the filenames for the region. Working code (modulo error handling and long paths): use std::{ffi::OsString, mem::{size_of, MaybeUninit}, os::windows::ffi::OsStringExt, path::Path};
use winapi::{
shared::{minwindef::{HMODULE, LPVOID}, ntdef::NULL},
um::{
libloaderapi::GetModuleFileNameW, memoryapi::VirtualQuery, sysinfoapi::GetSystemInfo,
winnt::MEMORY_BASIC_INFORMATION,
},
};
unsafe fn print_loaded_images() {
let mut system_info = MaybeUninit::zeroed().assume_init();
GetSystemInfo(&mut system_info);
let mut buf: Vec<u16> = Vec::with_capacity(1000);
let mut memory_basic_info = MaybeUninit::zeroed().assume_init();
let start = system_info.lpMinimumApplicationAddress;
let end = system_info.lpMaximumApplicationAddress;
let mut base = start;
while base < end {
VirtualQuery(base, &mut memory_basic_info, size_of::<MEMORY_BASIC_INFORMATION>());
if base != NULL && memory_basic_info.AllocationBase == base {
let len = GetModuleFileNameW(base as HMODULE, buf.as_mut_ptr(), 1000) as usize;
if len > 0 {
buf.set_len(len);
let filename = OsString::from_wide(&buf[..]);
let path = Path::new(&filename);
println!("base: {:x} module: {} ", base as usize, path.display());
}
}
base = ((base as usize) + memory_basic_info.RegionSize) as LPVOID;
}
}
fn main() {
unsafe { print_loaded_images() };
}And in your [dependencies.winapi]
version = "0.3.8"
features = ["libloaderapi", "memoryapi", "sysinfoapi"] |
|
Whoa that's perfect, thanks @lzybkr! One question I'd have for that, how does the call to In any case I can look to implement this at least for the gimli feature on the |
|
And another follow-up question. I've got an initial commit implementing this in the I'm getting some odd behavior, though, and was wondering if other windows folks knew what might be going on. I'm reliably seeing a backtrace for everything in the procedural macro itself: procedural macro's stack framesWhat's odd though is that the compiler sometimes works but sometimes doesn't: rest of backtrace, workingrest of backtrace, not workingMy guess here is that there's something to do with ASLR and/or some randomization of the address space. Currently that's not accounted for in the MinGW implementation of In the "not working" backtrace, for example, the address 0x2898cbd (frame 23) is detected as being inside of Is there a way to detect at runtime the "offset" of where a DLL is loaded? I'm basically trying to figure out the rough equivalent of |
|
You can use GetModuleHandleW to get the base address of a DLL or GetModuleHandleExW if you want to find the DLL that a certain address is contained in. The |
|
Regarding the cast - I don't think it's documented but seems unlikely to change. After a bit of digging, I think the officially supported api (and maybe slightly more reliable in multi-threaded code) uses the toolhelp api - here is an example in C++: https://docs.microsoft.com/en-us/windows/win32/toolhelp/traversing-the-module-list (though the sample should probably use Module32FirstW and Module32NextW. |
|
@alexcrichton ASLR is still not enabled for windows-gnu: #16514
|
alexcrichton
added a commit
to rust-lang/backtrace-rs
that referenced
this issue
Jun 10, 2020
This commit uses the information and APIs learned from rust-lang/rust#71060 to implement support for dynamic libraries on MinGW. Previously symbols only worked if they came from the main executable, but now the process's list of dynamic libraries are also searched so we can symbolicate, for example, symbols in the compiler which come from loaded libraries rather than the main executable.
alexcrichton
added a commit
to rust-lang/backtrace-rs
that referenced
this issue
Jun 10, 2020
This commit uses the information and APIs learned from rust-lang/rust#71060 to implement support for dynamic libraries on MinGW. Previously symbols only worked if they came from the main executable, but now the process's list of dynamic libraries are also searched so we can symbolicate, for example, symbols in the compiler which come from loaded libraries rather than the main executable.
alexcrichton
added a commit
to rust-lang/backtrace-rs
that referenced
this issue
Jun 10, 2020
This commit uses the information and APIs learned from rust-lang/rust#71060 to implement support for dynamic libraries on MinGW. Previously symbols only worked if they came from the main executable, but now the process's list of dynamic libraries are also searched so we can symbolicate, for example, symbols in the compiler which come from loaded libraries rather than the main executable.
|
Ok I've whipped up rust-lang/backtrace-rs#345 which I believe works at least for the
|
alexcrichton
added a commit
to rust-lang/backtrace-rs
that referenced
this issue
Jun 10, 2020
This commit uses the information and APIs learned from rust-lang/rust#71060 to implement support for dynamic libraries on MinGW. Previously symbols only worked if they came from the main executable, but now the process's list of dynamic libraries are also searched so we can symbolicate, for example, symbols in the compiler which come from loaded libraries rather than the main executable.
|
Well, it turns out libbacktrace itself works fine for MinGW. Libgcc has a bug in
|
|
Removing nomination as it was discussed during today's meeting. |
Discussed in today's T-compiler meeting No one had a terribly strong opinion. The team leads both agreed that for the short-term, if its easy to change the compiler to use the |
6 tasks
alexcrichton
added a commit
to alexcrichton/rust
that referenced
this issue
Jul 17, 2020
This commit is a proof-of-concept for switching the standard library's backtrace symbolication mechanism on most platforms from libbacktrace to gimli. The standard library's support for `RUST_BACKTRACE=1` requires in-process parsing of object files and DWARF debug information to interpret it and print the filename/line number of stack frames as part of a backtrace. Historically this support in the standard library has come from a library called "libbacktrace". The libbacktrace library seems to have been extracted from gcc at some point and is written in C. We've had a lot of issues with libbacktrace over time, unfortunately, though. The library does not appear to be actively maintained since we've had patches sit for months-to-years without comments. We have discovered a good number of soundness issues with the library itself, both when parsing valid DWARF as well as invalid DWARF. This is enough of an issue that the libs team has previously decided that we cannot feed untrusted inputs to libbacktrace. This also doesn't take into account the portability of libbacktrace which has been difficult to manage and maintain over time. While possible there are lots of exceptions and it's the main C dependency of the standard library right now. For years it's been the desire to switch over to a Rust-based solution for symbolicating backtraces. It's been assumed that we'll be using the Gimli family of crates for this purpose, which are targeted at safely and efficiently parsing DWARF debug information. I've been working recently to shore up the Gimli support in the `backtrace` crate. As of a few weeks ago the `backtrace` crate, by default, uses Gimli when loaded from crates.io. This transition has gone well enough that I figured it was time to start talking seriously about this change to the standard library. This commit is a preview of what's probably the best way to integrate the `backtrace` crate into the standard library with the Gimli feature turned on. While today it's used as a crates.io dependency, this commit switches the `backtrace` crate to a submodule of this repository which will need to be updated manually. This is not done lightly, but is thought to be the best solution. The primary reason for this is that the `backtrace` crate needs to do some pretty nontrivial filesystem interactions to locate debug information. Working without `std::fs` is not an option, and while it might be possible to do some sort of trait-based solution when prototyped it was found to be too unergonomic. Using a submodule allows the `backtrace` crate to build as a submodule of the `std` crate itself, enabling it to use `std::fs` and such. Otherwise this adds new dependencies to the standard library. This step requires extra attention because this means that these crates are now going to be included with all Rust programs by default. It's important to note, however, that we're already shipping libbacktrace with all Rust programs by default and it has a bunch of C code implementing all of this internally anyway, so we're basically already switching already-shipping functionality to Rust from C. * `object` - this crate is used to parse object file headers and contents. Very low-level support is used from this crate and almost all of it is disabled. Largely we're just using struct definitions as well as convenience methods internally to read bytes and such. * `addr2line` - this is the main meat of the implementation for symbolication. This crate depends on `gimli` for DWARF parsing and then provides interfaces needed by the `backtrace` crate to turn an address into a filename / line number. This crate is actually pretty small (fits in a single file almost!) and mirrors most of what `dwarf.c` does for libbacktrace. * `miniz_oxide` - the libbacktrace crate transparently handles compressed debug information which is compressed with zlib. This crate is used to decompress compressed debug sections. * `gimli` - not actually used directly, but a dependency of `addr2line`. * `adler32`- not used directly either, but a dependency of `miniz_oxide`. The goal of this change is to improve the safety of backtrace symbolication in the standard library, especially in the face of possibly malformed DWARF debug information. Even to this day we're still seeing segfaults in libbacktrace which could possibly become security vulnerabilities. This change should almost entirely eliminate this possibility whilc also paving the way forward to adding more features like split debug information. Some references for those interested are: * Original addition of libbacktrace - rust-lang#12602 * OOM with libbacktrace - rust-lang#24231 * Backtrace failure due to use of uninitialized value - rust-lang#28447 * Possibility to feed untrusted data to libbacktrace - rust-lang#21889 * Soundness fix for libbacktrace - rust-lang#33729 * Crash in libbacktrace - rust-lang#39468 * Support for macOS, never merged - ianlancetaylor/libbacktrace#2 * Performance issues with libbacktrace - rust-lang#29293, rust-lang#37477 * Update procedure is quite complicated due to how many patches we need to carry - rust-lang#50955 * Libbacktrace doesn't work on MinGW with dynamic libs - rust-lang#71060 * Segfault in libbacktrace on macOS - rust-lang#71397 Switching to Rust will not make us immune to all of these issues. The crashes are expected to go away, but correctness and performance may still have bugs arise. The gimli and `backtrace` crates, however, are actively maintained unlike libbacktrace, so this should enable us to at least efficiently apply fixes as situations come up.
bors
added a commit
to rust-lang-ci/rust
that referenced
this issue
Jul 18, 2020
…Simulacrum std: Switch from libbacktrace to gimli This commit is a proof-of-concept for switching the standard library's backtrace symbolication mechanism on most platforms from libbacktrace to gimli. The standard library's support for `RUST_BACKTRACE=1` requires in-process parsing of object files and DWARF debug information to interpret it and print the filename/line number of stack frames as part of a backtrace. Historically this support in the standard library has come from a library called "libbacktrace". The libbacktrace library seems to have been extracted from gcc at some point and is written in C. We've had a lot of issues with libbacktrace over time, unfortunately, though. The library does not appear to be actively maintained since we've had patches sit for months-to-years without comments. We have discovered a good number of soundness issues with the library itself, both when parsing valid DWARF as well as invalid DWARF. This is enough of an issue that the libs team has previously decided that we cannot feed untrusted inputs to libbacktrace. This also doesn't take into account the portability of libbacktrace which has been difficult to manage and maintain over time. While possible there are lots of exceptions and it's the main C dependency of the standard library right now. For years it's been the desire to switch over to a Rust-based solution for symbolicating backtraces. It's been assumed that we'll be using the Gimli family of crates for this purpose, which are targeted at safely and efficiently parsing DWARF debug information. I've been working recently to shore up the Gimli support in the `backtrace` crate. As of a few weeks ago the `backtrace` crate, by default, uses Gimli when loaded from crates.io. This transition has gone well enough that I figured it was time to start talking seriously about this change to the standard library. This commit is a preview of what's probably the best way to integrate the `backtrace` crate into the standard library with the Gimli feature turned on. While today it's used as a crates.io dependency, this commit switches the `backtrace` crate to a submodule of this repository which will need to be updated manually. This is not done lightly, but is thought to be the best solution. The primary reason for this is that the `backtrace` crate needs to do some pretty nontrivial filesystem interactions to locate debug information. Working without `std::fs` is not an option, and while it might be possible to do some sort of trait-based solution when prototyped it was found to be too unergonomic. Using a submodule allows the `backtrace` crate to build as a submodule of the `std` crate itself, enabling it to use `std::fs` and such. Otherwise this adds new dependencies to the standard library. This step requires extra attention because this means that these crates are now going to be included with all Rust programs by default. It's important to note, however, that we're already shipping libbacktrace with all Rust programs by default and it has a bunch of C code implementing all of this internally anyway, so we're basically already switching already-shipping functionality to Rust from C. * `object` - this crate is used to parse object file headers and contents. Very low-level support is used from this crate and almost all of it is disabled. Largely we're just using struct definitions as well as convenience methods internally to read bytes and such. * `addr2line` - this is the main meat of the implementation for symbolication. This crate depends on `gimli` for DWARF parsing and then provides interfaces needed by the `backtrace` crate to turn an address into a filename / line number. This crate is actually pretty small (fits in a single file almost!) and mirrors most of what `dwarf.c` does for libbacktrace. * `miniz_oxide` - the libbacktrace crate transparently handles compressed debug information which is compressed with zlib. This crate is used to decompress compressed debug sections. * `gimli` - not actually used directly, but a dependency of `addr2line`. * `adler32`- not used directly either, but a dependency of `miniz_oxide`. The goal of this change is to improve the safety of backtrace symbolication in the standard library, especially in the face of possibly malformed DWARF debug information. Even to this day we're still seeing segfaults in libbacktrace which could possibly become security vulnerabilities. This change should almost entirely eliminate this possibility whilc also paving the way forward to adding more features like split debug information. Some references for those interested are: * Original addition of libbacktrace - rust-lang#12602 * OOM with libbacktrace - rust-lang#24231 * Backtrace failure due to use of uninitialized value - rust-lang#28447 * Possibility to feed untrusted data to libbacktrace - rust-lang#21889 * Soundness fix for libbacktrace - rust-lang#33729 * Crash in libbacktrace - rust-lang#39468 * Support for macOS, never merged - ianlancetaylor/libbacktrace#2 * Performance issues with libbacktrace - rust-lang#29293, rust-lang#37477 * Update procedure is quite complicated due to how many patches we need to carry - rust-lang#50955 * Libbacktrace doesn't work on MinGW with dynamic libs - rust-lang#71060 * Segfault in libbacktrace on macOS - rust-lang#71397 Switching to Rust will not make us immune to all of these issues. The crashes are expected to go away, but correctness and performance may still have bugs arise. The gimli and `backtrace` crates, however, are actively maintained unlike libbacktrace, so this should enable us to at least efficiently apply fixes as situations come up. --- I want to note that my purpose for creating a PR here is to start a conversation about this. I think that all the various pieces are in place that this is compelling enough that I think this transition should be talked about seriously. There are a number of items which still need to be addressed before actually merging this PR, however: * [ ] `gimli` needs to be published to crates.io * [ ] `addr2line` needs a publish * [ ] `miniz_oxide` needs a publish * [ ] Tests probably shouldn't recommend the `gimli` crate's traits for implementing * [ ] The `backtrace` crate's branch changes need to be merged to the master branch (rust-lang/backtrace-rs#349) * [ ] The support for `libbacktrace` on some platforms needs to be audited to see if we should support more strategies in the gimli implementation - rust-lang/backtrace-rs#325, rust-lang/backtrace-rs#326, rust-lang/backtrace-rs#350, rust-lang/backtrace-rs#351 Most of the merging/publishing I'm not actively pushing on right now. It's a bit wonky for crates to support libstd so I'm holding off on pulling the trigger everywhere until there's a bit more discussion about how to go through with this. Namely rust-lang/backtrace-rs#349 I'm going to hold off merging until we decide to go through with the submodule strategy. In any case this is a pretty major change, so I suspect that the compiler team is likely going to be interested in this. I don't mean to force changes by dumping a bunch of code by any means. Integration of external crates into the standard library is so difficult I wanted to have a proof-of-concept to review while talking about whether to do this at all (hence the PR), but I'm more than happy to follow any processes needed to merge this. I must admit though that I'm not entirely sure myself at this time what the process would be to decide to merge this, so I'm hoping others can help me figure that out!
|
Example of beautiful ICE backtrace on current nightly with gimli 🚀: @alexcrichton should we keep this issue open considering #73441 will be probably reverted and relanded? |
|
I think let's go ahead and close and if that's backed out we can reopen. |
|
Ok PR ended up being backed out. |
alexcrichton
added a commit
to alexcrichton/rust
that referenced
this issue
Jul 28, 2020
This commit is a proof-of-concept for switching the standard library's backtrace symbolication mechanism on most platforms from libbacktrace to gimli. The standard library's support for `RUST_BACKTRACE=1` requires in-process parsing of object files and DWARF debug information to interpret it and print the filename/line number of stack frames as part of a backtrace. Historically this support in the standard library has come from a library called "libbacktrace". The libbacktrace library seems to have been extracted from gcc at some point and is written in C. We've had a lot of issues with libbacktrace over time, unfortunately, though. The library does not appear to be actively maintained since we've had patches sit for months-to-years without comments. We have discovered a good number of soundness issues with the library itself, both when parsing valid DWARF as well as invalid DWARF. This is enough of an issue that the libs team has previously decided that we cannot feed untrusted inputs to libbacktrace. This also doesn't take into account the portability of libbacktrace which has been difficult to manage and maintain over time. While possible there are lots of exceptions and it's the main C dependency of the standard library right now. For years it's been the desire to switch over to a Rust-based solution for symbolicating backtraces. It's been assumed that we'll be using the Gimli family of crates for this purpose, which are targeted at safely and efficiently parsing DWARF debug information. I've been working recently to shore up the Gimli support in the `backtrace` crate. As of a few weeks ago the `backtrace` crate, by default, uses Gimli when loaded from crates.io. This transition has gone well enough that I figured it was time to start talking seriously about this change to the standard library. This commit is a preview of what's probably the best way to integrate the `backtrace` crate into the standard library with the Gimli feature turned on. While today it's used as a crates.io dependency, this commit switches the `backtrace` crate to a submodule of this repository which will need to be updated manually. This is not done lightly, but is thought to be the best solution. The primary reason for this is that the `backtrace` crate needs to do some pretty nontrivial filesystem interactions to locate debug information. Working without `std::fs` is not an option, and while it might be possible to do some sort of trait-based solution when prototyped it was found to be too unergonomic. Using a submodule allows the `backtrace` crate to build as a submodule of the `std` crate itself, enabling it to use `std::fs` and such. Otherwise this adds new dependencies to the standard library. This step requires extra attention because this means that these crates are now going to be included with all Rust programs by default. It's important to note, however, that we're already shipping libbacktrace with all Rust programs by default and it has a bunch of C code implementing all of this internally anyway, so we're basically already switching already-shipping functionality to Rust from C. * `object` - this crate is used to parse object file headers and contents. Very low-level support is used from this crate and almost all of it is disabled. Largely we're just using struct definitions as well as convenience methods internally to read bytes and such. * `addr2line` - this is the main meat of the implementation for symbolication. This crate depends on `gimli` for DWARF parsing and then provides interfaces needed by the `backtrace` crate to turn an address into a filename / line number. This crate is actually pretty small (fits in a single file almost!) and mirrors most of what `dwarf.c` does for libbacktrace. * `miniz_oxide` - the libbacktrace crate transparently handles compressed debug information which is compressed with zlib. This crate is used to decompress compressed debug sections. * `gimli` - not actually used directly, but a dependency of `addr2line`. * `adler32`- not used directly either, but a dependency of `miniz_oxide`. The goal of this change is to improve the safety of backtrace symbolication in the standard library, especially in the face of possibly malformed DWARF debug information. Even to this day we're still seeing segfaults in libbacktrace which could possibly become security vulnerabilities. This change should almost entirely eliminate this possibility whilc also paving the way forward to adding more features like split debug information. Some references for those interested are: * Original addition of libbacktrace - rust-lang#12602 * OOM with libbacktrace - rust-lang#24231 * Backtrace failure due to use of uninitialized value - rust-lang#28447 * Possibility to feed untrusted data to libbacktrace - rust-lang#21889 * Soundness fix for libbacktrace - rust-lang#33729 * Crash in libbacktrace - rust-lang#39468 * Support for macOS, never merged - ianlancetaylor/libbacktrace#2 * Performance issues with libbacktrace - rust-lang#29293, rust-lang#37477 * Update procedure is quite complicated due to how many patches we need to carry - rust-lang#50955 * Libbacktrace doesn't work on MinGW with dynamic libs - rust-lang#71060 * Segfault in libbacktrace on macOS - rust-lang#71397 Switching to Rust will not make us immune to all of these issues. The crashes are expected to go away, but correctness and performance may still have bugs arise. The gimli and `backtrace` crates, however, are actively maintained unlike libbacktrace, so this should enable us to at least efficiently apply fixes as situations come up.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Backtraces work fine when using
panic!()inside crate but they don't whenrustcICEs.Meta
Backtrace from building Rust with `debuginfo-level = 1`
Backtrace from nightly
The text was updated successfully, but these errors were encountered: