-
Notifications
You must be signed in to change notification settings - Fork 29.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Failing test: test-inspector-multisession-ws #32383
Comments
@nodejs/inspector |
This is with V8 lkgr (canary branch of this repo) |
FYI, I'm looking into this (couldn't find any PRs referencing this issue, so I assume no one fixed it yet). The child process (the process being inspected) is segfaulting when it receives a // script.js
const { Session } = require('inspector');
const session = new Session();
let done = false;
const interval = setInterval(() => {
process._rawDebug('hey oh');
if (done)
clearInterval(interval);
}, 1000);
session.on('Debugger.paused', () => {
done = true;
});
session.connect();
session.post('Debugger.enable');
console.log('Ready'); // connect.js
'use strict'
const http = require("http");
const WebSocket = require('ws');
function usage() {
console.error('USAGE: node index.js <PID>')
process.exit(1)
}
if (process.argv.length !== 3) usage();
const pid = Number(process.argv[2])
if (pid === NaN) usage();
let first = true;
function connectToInspector(url) {
const ws = new WebSocket(url);
ws.on('message', function incoming(data) {
const res = JSON.parse(data);
if (first && res.method == "Debugger.paused") {
first = false;
ws.send(JSON.stringify({"method": "Debugger.resume", "id": 4}));
}
if (res.method == "Debugger.resumed")
ws.send(JSON.stringify({"method": "Debugger.pause", "id": 3}));
if (res.method != "Debugger.scriptParsed")
console.log(res);
});
ws.on('open', function open() {
ws.send(JSON.stringify({"method": "Debugger.enable", "id": 1}));
ws.send(JSON.stringify({"method": "Runtime.runIfWaitingForDebugger", "id": 2}));
});
}
function getWebSocketPath(cb) {
http.get("http://localhost:9229/json", (resp) => {
let data = '';
resp.on('data', (chunk) => {
data += chunk;
});
resp.on('end', () => cb(JSON.parse(data)[0].webSocketDebuggerUrl));
})
}
process.kill(pid, 'SIGUSR1');
getWebSocketPath(connectToInspector); $ node --inspect-brk script.js &
[1] 13214
Debugger listening on ws://127.0.0.1:9229/6680ec92-a986-4a8c-8f2f-0daee5deabfb
For help, see: https://nodejs.org/en/docs/inspector
$ node connect.js $(pgrep node)
Debugger attached.
{
id: 1,
result: { debuggerId: '-5779516113788299134.3822508824226157107' }
}
{ id: 2, result: {} }
{
method: 'Debugger.paused',
params: {
callFrames: [
[Object], [Object],
[Object], [Object],
[Object], [Object],
[Object]
],
reason: 'Break on start',
hitBreakpoints: []
}
}
{ id: 4, result: {} }
{ method: 'Debugger.resumed', params: {} }
{ id: 3, result: {} }
[1] + 14390 segmentation fault (core dumped) /home/mmarchini/workspace/nodejs/node-v8/node --inspect-brk fooo.js Stack from core dump:
Interestingly enough, this only happens with I'll keep investigating and share any results I find. |
My current theory is that we're hitting pause right after a script is loaded, parsed, maybe even compiled, and pushed onto the stack, but it's not quite ready to run yet. Why this is happening so frequently? When Why this is not happening on master? No idea Is this a bug on V8, Node.js or in our test? I'm inclined to say V8, but want to look a bit further before opening a bug anywhere. There are a few fixes we can apply to the test if we want to unblock the upgrade and investigate this issue in parallel. Let me know if that's something you want, I can send a patch. Notes I took while debugging, mostrly unstrucutred, unfiltered, and unrevisedIf I add a small delay between Starting the process with The debug build is hitting a DCHECK on the same function the Release build is crashing (here). We're likely reaching that code while in an inconsistent state. Here's the (supposedly) stack trace on the DCHECK when it fails (printed using gdbinit's jst):
Workers, huh? I wonder what that's doing there (and why our code is in the middle of the stack). Ok, let's try a smaller piece of code: let done = false;
function lala() {
setInterval(lala, 150);
}
lala();
console.log('Ready');
Ok, now we are in TTY, apparently inside In both cases, the type of require("tty");
let done = false;
function lala() {
setInterval(lala, 150);
}
lala();
console.log('Ready'); Well, this works. Let's try the other way around. The code above should break on Chrome DevTools:
require("./require-me"); Well, Chrome DevTools worked. I tried the same on a Node.js module and got the same result. So, looking back at the output of
Notice the
This gives me an idea to break // crash-without-brk.js
const { Session } = require('inspector');
const session = new Session();
let done = false;
function lala() {
setInterval(lala, 150);
if (done)
console.log('Ready');
}
lala();
session.on('Debugger.paused', () => {
done = true;
});
session.connect();
session.post('Debugger.enable'); And boom! $ node --inspect crash-without-brk.js
# after calling the websocket script
#
# Fatal error in ../../deps/v8/src/debug/debug-scopes.cc, line 452
# Debug check failed: current_scope_->NeedsContext() implies context_->IsFunctionContext() || context_->IsDebugEvaluateContext().
#
#
#
#FailureMessage Object: 0x7ffee58c22f0
1: 0x56523d23cad4 node::DumpBacktrace(_IO_FILE*) [/home/mmarchini/workspace/nodejs/node-v8/out/Debug/node]
2: 0x56523d3c6f1f [/home/mmarchini/workspace/nodejs/node-v8/out/Debug/node]
3: 0x56523d3c6f43 [/home/mmarchini/workspace/nodejs/node-v8/out/Debug/node]
4: 0x56523ee3536b V8_Fatal(char const*, int, char const*, ...) [/home/mmarchini/workspace/nodejs/node-v8/out/Debug/node]
5: 0x56523ee3539b [/home/mmarchini/workspace/nodejs/node-v8/out/Debug/node]
6: 0x56523d79f48e v8::internal::ScopeIterator::Type() const [/home/mmarchini/workspace/nodejs/node-v8/out/Debug/node]
7: 0x56523d79d168 v8::internal::DebugScopeIterator::DebugScopeIterator(v8::internal::Isolate*, v8::internal::FrameInspector*) [/home/mmarchini/workspace/nodejs/node-v8/out/Debug/node]
8: 0x56523d7a8878 v8::internal::DebugStackTraceIterator::GetScopeIterator() const [/home/mmarchini/workspace/nodejs/node-v8/out/Debug/node]
9: 0x56523e11736b v8_inspector::V8DebuggerAgentImpl::currentCallFrames(std::unique_ptr<std::vector<std::unique_ptr<v8_inspector::protocol::Debugger::CallFrame, std::default_delete<v8_inspector::protocol::Debugger::CallFrame> >, std::allocator<std::unique_ptr<v8_inspector::
protocol::Debugger::CallFrame, std::default_delete<v8_inspector::protocol::Debugger::CallFrame> > > >, std::default_delete<std::vector<std::unique_ptr<v8_inspector::protocol::Debugger::CallFrame, std::default_delete<v8_inspector::protocol::Debugger::CallFrame> >, std::alloc
ator<std::unique_ptr<v8_inspector::protocol::Debugger::CallFrame, std::default_delete<v8_inspector::protocol::Debugger::CallFrame> > > > > >*) [/home/mmarchini/workspace/nodejs/node-v8/out/Debug/node]
10: 0x56523e11ee77 v8_inspector::V8DebuggerAgentImpl::didPause(int, v8::Local<v8::Value>, std::vector<int, std::allocator<int> > const&, v8::debug::ExceptionType, bool, bool, bool) [/home/mmarchini/workspace/nodejs/node-v8/out/Debug/node]
11: 0x56523e1063d0 [/home/mmarchini/workspace/nodejs/node-v8/out/Debug/node]
12: 0x56523e128d01 v8_inspector::V8InspectorImpl::forEachSession(int, std::function<void (v8_inspector::V8InspectorSessionImpl*)> const&) [/home/mmarchini/workspace/nodejs/node-v8/out/Debug/node]
13: 0x56523e1097ff v8_inspector::V8Debugger::handleProgramBreak(v8::Local<v8::Context>, v8::Local<v8::Value>, std::vector<int, std::allocator<int> > const&, v8::debug::ExceptionType, bool) [/home/mmarchini/workspace/nodejs/node-v8/out/Debug/node]
14: 0x56523d7bf06d v8::internal::Debug::OnDebugBreak(v8::internal::Handle<v8::internal::FixedArray>) [/home/mmarchini/workspace/nodejs/node-v8/out/Debug/node]
15: 0x56523d7bf24b v8::internal::Debug::HandleDebugBreak(v8::internal::IgnoreBreakMode) [/home/mmarchini/workspace/nodejs/node-v8/out/Debug/node]
16: 0x56523d5850eb v8::debug::BreakRightNow(v8::Isolate*) [/home/mmarchini/workspace/nodejs/node-v8/out/Debug/node]
17: 0x56523d854939 v8::internal::Isolate::InvokeApiInterruptCallbacks() [/home/mmarchini/workspace/nodejs/node-v8/out/Debug/node]
18: 0x56523d881012 v8::internal::StackGuard::HandleInterrupts() [/home/mmarchini/workspace/nodejs/node-v8/out/Debug/node]
19: 0x56523de7aafb [/home/mmarchini/workspace/nodejs/node-v8/out/Debug/node]
20: 0x56523de7ae8e v8::internal::Runtime_StackGuard(int, unsigned long*, v8::internal::Isolate*) [/home/mmarchini/workspace/nodejs/node-v8/out/Debug/node]
21: 0x56523e48ccbf [/home/mmarchini/workspace/nodejs/node-v8/out/Debug/node]
[1] 11135 illegal hardware instruction (core dumped) /home/mmarchini/workspace/nodejs/node-v8/out/Debug/node --inspect crash-without-brk.js This is all accetonal evidence, but it leads me to believe the issue is manifesting when we call pause and a recently loaded script is in an intermediate state (it's not running yet, but it was at least parsed, probably compiled and loaded on the stack). |
@mmarchini thanks a lot for looking into this! |
Likely caused by this change: https://chromium-review.googlesource.com/c/v8/v8/+/1876821. Now the NativeContext has a SCRIPT_SCOPE attached to it instead of a FUNCTION_SCOPE. Will keep digging, I think we might be using the NativeContext in an incompatible way compared to the new behavior. |
Is this test flaky ? it's not seeing in node-v8 master. |
Yes, but the failing is legit
Do you mean |
yeap, only 3 test failed, but this one is not one of them. |
Still happening: $ ./node
[test] Connecting to a child Node process
[test] Testing /json/list
[test] Connecting to a child Node process
[test] Testing /json/list
[err] Debugger listening on ws://127.0.0.1:44189/05bc75ec-1ceb-4da4-a207-53573d3b400d
[err] For help, see: https://nodejs.org/en/docs/inspector
[err]
[err] Debugger attached.
[err] Debugger attached.
[err]
[test] Breaking in code and verifying events are fired
Timed out waiting for matching notification (SessionA paused))
1
I also used one of the scripts described in https://github.com/nodejs/node-v8/issues/120#issuecomment-589939128, and it is still segfaulting. I'm inclined to say this is a bug on V8. I'm trying to come up with a test on V8 for this, but writing a custom inspector test is somewhat investing, so it's taking some time. I was able to confirm (using bpftrace and some debugging printf's) that the condition for the segfault to happen is:
The most likely scenario for this to happen is when we try to pause while requiring an internal module, although in theory we can probably trigger it with |
It also failed on the latest Jenkins run: https://ci.nodejs.org/job/node-test-commit-linux/33109/nodes=ubuntu1804-64/consoleFull |
Weird github CI not reporting this error. Any clue ? @cclauss |
The test is super flaky (the |
Is it possible we biselect v8 commit to find the problem ? |
The commit where it started is likely https://chromium-review.googlesource.com/c/v8/v8/+/1876821, but I haven't bisected to double check (since I don't have a test on V8 yet, would need to bisect on node-v8, which is too much effort). I think I'm close to have a repro on V8 though. |
Ok, got a repro on V8. I had to copy an entire script there (tty.js), so I'm not sure what exactly is triggering the bug. Will start to refactor the test to see where the problem is. |
V8 minimal repro: https://chromium-review.googlesource.com/c/v8/v8/+/2080072 |
This is such an edge case I'm wondering if we should mark the test as flaky (and maybe add a test specific for this bug on
|
I am on adding it to |
FWIW this bug is unrelated to multi-session. It was a coincidence this happened on this test. |
Sorry for miss that part. Is there |
It's a folder there (https://github.com/nodejs/node/tree/master/test/known_issues, I only found out about it a few days ago). Tests there are expected to always fail, so |
Also, for anyone interested, v8 crashes in here [err] #
[err] # Fatal error in ../../deps/v8/src/debug/debug-scopes.cc, line 454
[err] # Debug check failed: current_scope_->NeedsContext() implies context_->IsFunctionContext() || context_->IsDebugEvaluateContext(). |
It failed rather consistently, but it would succeed occasionally. |
Ref: https://github.com/nodejs/node-v8/issues/120 Signed-off-by: Matheus Marchini <mmarchini@netflix.com>
Ref: https://github.com/nodejs/node-v8/issues/120 Signed-off-by: Matheus Marchini <mmarchini@netflix.com>
Ref: https://github.com/nodejs/node-v8/issues/120 Signed-off-by: Matheus Marchini <mmarchini@netflix.com>
Ref: https://github.com/nodejs/node-v8/issues/120 Signed-off-by: Matheus Marchini <mmarchini@netflix.com>
Ref: https://github.com/nodejs/node-v8/issues/120 Signed-off-by: Matheus Marchini <mmarchini@netflix.com>
Ref: https://github.com/nodejs/node-v8/issues/120 Signed-off-by: Matheus Marchini <mmarchini@netflix.com>
Ref: https://github.com/nodejs/node-v8/issues/120 Signed-off-by: Matheus Marchini <mmarchini@netflix.com>
Ref: https://github.com/nodejs/node-v8/issues/120 Signed-off-by: Matheus Marchini <mmarchini@netflix.com>
Ref: https://github.com/nodejs/node-v8/issues/120 Signed-off-by: Matheus Marchini <mmarchini@netflix.com>
Ref: https://github.com/nodejs/node-v8/issues/120 Signed-off-by: Matheus Marchini <mmarchini@netflix.com>
We should open a tracking issue on nodejs/node, since the underlying bug still exists on V8 |
Move this to issue to |
I thought opening a new issue made more sense, but @targos was faster than me :) |
The text was updated successfully, but these errors were encountered: