Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix LLDB test failures after latest dependency version updates. #1813

Open
cfallin opened this issue Jun 3, 2020 · 15 comments
Open

Fix LLDB test failures after latest dependency version updates. #1813

cfallin opened this issue Jun 3, 2020 · 15 comments
Labels
bug Incorrect behavior in the current implementation that needs fixing

Comments

@cfallin
Copy link
Member

cfallin commented Jun 3, 2020

After #1805, we're seeing test failures like the following in debug::lldb::test_debug_dwarf_ptr:

``` test debug::lldb::test_debug_dwarf_ptr ... FAILED

failures:

---- debug::lldb::test_debug_dwarf_ptr stdout ----
thread 'main' panicked at 'didn't pass check > (lldb) target create "/home/runner/work/wasmtime/wasmtime/target/debug/wasmtime"

Current executable set to '/home/runner/work/wasmtime/wasmtime/target/debug/wasmtime' (x86_64).
(lldb) settings set -- target.run-args "-g" "--opt-level" "0" "tests/all/debug/testsuite/reverse-str.wasm"
(lldb) command source -s 0 '/tmp/.tmpmh81yA'
Executing commands in '/tmp/.tmpmh81yA'.
(lldb) b reverse-str.c:9
Breakpoint 1: no locations (pending).
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Matched #0: \bBreakpoint 1: no locations (pending)
WARNING: Unable to resolve breakpoint to any actual locations.
(lldb) r
1 location added to breakpoint 1
Process 14667 stopped

Process 14667 launched: '/home/runner/work/wasmtime/wasmtime/target/debug/wasmtime' (x86_64)
(lldb) p __vmctx->set(),&*s
', tests/all/debug/lldb.rs:52:5

</details>

We suspect the version-bump of some dependency caused this, though we aren't yet sure why (and we cannot reproduce locally). We will disable the test for now but we need to investigate and re-enable.
cfallin added a commit to cfallin/wasmtime that referenced this issue Jun 3, 2020
@cfallin
Copy link
Member Author

cfallin commented Jun 3, 2020

We're temporarily avoiding CI failures by pinning versions in Cargo.lock in #1811, but we should resolve the underlying issue and the update the lock file to the latest versions.

@cfallin cfallin added the bug Incorrect behavior in the current implementation that needs fixing label Jun 3, 2020
@cfallin
Copy link
Member Author

cfallin commented Jun 3, 2020

cc @yurydelendik

@alexcrichton
Copy link
Member

With a green master it's clear that cargo update -p backtrace is causing the regression here. I can reproduce this locally, and the problem is that LLDB prints:

error: Can't run the expression locally: Interpreter doesn't handle one of the expression's opcodes

when we try to execute

p __vmctx->set(),&*s

I'm not really sure how the backtrace crate is related to this. Even the gimli/addr2line/object/etc updates don't seem to affect wasmtime since they're all on versions that wasmtime itself isn't using, only the backtrace crate is using them.

@yurydelendik do you know where that error message may be coming from?

@yurydelendik
Copy link
Contributor

@alexcrichton "backtrace" now indirectly depends on "gimli 0.21", I'm thinking it is what is giving an issue. I'm working on upgrading our "gimli 0.20" atm, though not sure why it is giving "Interpreter doesn't handle one of the expression's opcodes".

@alexcrichton
Copy link
Member

I'm not sure why the gimli 0.21 dep would cause any issues though? That's exclusively used by the backtrace crate (everything else in wasmtime is using 0.20), so is it like it's mere presence is causing issues?

@yurydelendik
Copy link
Contributor

I'm not sure why the gimli 0.21 dep would cause any issues though?

Me neither. But I would like to upgrade it anyway, but it is not easy. Too much API refactoring happened between 0.20 -> 0.21 .

@yurydelendik
Copy link
Contributor

yurydelendik commented Jun 3, 2020

do you know where that error message may be coming from?

It is likely LLDB -- it cannot decode generated by wasmtime DWARF expression

P.S. it is reason I'm connecting it with gimli

@yurydelendik
Copy link
Contributor

It is limited to Linux platforms. Bisect points for rust-lang/backtrace-rs@5ad0b76

@alexcrichton
Copy link
Member

At this point it sounds like this is exposing a preexisting bug. The only real change in that commit which would affect this is that libbacktrace's C code is gone, and some new Rust code is pulled in. For these tests and LLDB I don't think any of that code is executed, but presumably the DWARF for wasmtime has changed, perhaps significantly, as a result.

Would it make sense to try to track this down not as a regression in the backtrace crate but instead as a possible bug in wasmtime's emission of dwarf information? (or LLDB's interpretation of it?)

@yurydelendik
Copy link
Contributor

yurydelendik commented Jun 4, 2020

@alexcrichton the problem is related to LLDB only (works on GDB). Running any p any_call() inside wasmtime debug session, without even JIT code, causes that too:

(lldb) b wasmtime.rs:63
(lldb) r
(lldb) p set_vmctx_memory(0)

Which fails with the same error.

Maybe one of the new modules contains unsupported DWARF data?

P.S. also works in release mode, e.g. cargo test --release test_debug_dwarf_ptr -- --ignored

@yurydelendik
Copy link
Contributor

yurydelendik commented Jun 4, 2020

Adding to Cargo.toml:

[profile.dev.package.backtrace]
debug = 0

allows test to past.


Tested with lldb-10 for Ubuntu -- it works there too. (It was working for lldb-9 on MacOSX, not on Ubunty)

If CI will start using lldb-10, the problem will go away.

@alexcrichton
Copy link
Member

Nice find! I'd be fine taking care of the CI bits if you want to check in that fix to Cargo.toml?

@yurydelendik
Copy link
Contributor

yurydelendik commented Jun 4, 2020

if you want to check in that fix to Cargo.toml?

Sorry, there are two solutions above: fix in Cargo.toml or lldb-10. Moving CI to lldb-10 will completely fix the issue (without Cargo.toml change).

@alexcrichton
Copy link
Member

So I thought LLDB 10 would be pretty easy to install from https://github.com/llvm/llvm-project/releases/tag/llvmorg-10.0.0, but unfortunately those tarballs don't contain lldb at all. They did for LLVM 9 though (?)

@fitzgen
Copy link
Member

fitzgen commented Jun 15, 2020

Note that #1832 is going to update backtrace, but is adding

# Cargo.toml
[profile.dev.package.backtrace]
debug = false # FIXME(#1813)

to temporarily work around these failures.

We should remove that when properly fixing this bug.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Incorrect behavior in the current implementation that needs fixing
Projects
None yet
Development

No branches or pull requests

4 participants