Skip to content
This repository has been archived by the owner on Apr 22, 2023. It is now read-only.

Segfault on FreeBSD #9326

Closed
sigmike opened this issue Mar 4, 2015 · 30 comments
Closed

Segfault on FreeBSD #9326

sigmike opened this issue Mar 4, 2015 · 30 comments
Assignees
Milestone

Comments

@sigmike
Copy link

sigmike commented Mar 4, 2015

I'm running this simple source:

var http = require("http");

http.createServer(function(request, response) {
  response.writeHead(200, {"Content-Type": "text/plain"});
  response.write("Hello World");
  response.end();
}).listen(5000);

It works fine until I hit Ctrl+C: it segfaults.

Here's the backtrace:

#0  0x000000080226f469 in swapcontext () from /lib/libthr.so.3
#1  0x000000080226f062 in sigaction () from /lib/libthr.so.3
#2  <signal handler called>
#3  0x00000008024f344a in _umtx_op () from /lib/libc.so.7
#4  0x00000008024e2c24 in sem_timedwait () from /lib/libc.so.7
#5  0x0000000000b53ef8 in v8::base::Semaphore::Wait ()
#6  0x00000000008c7e84 in v8::internal::SweeperThread::Run ()
#7  0x0000000000b5594d in v8::base::ThreadEntry ()
#8  0x000000080226a4f5 in pthread_create () from /lib/libthr.so.3
#9  0x0000000000000000 in ?? ()

I'm running FreeBSD 10.1-RELEASE with node installed from the ports.

If I run a script that contains only a console.log it doesn't crash when it exits.

@sigmike
Copy link
Author

sigmike commented Mar 4, 2015

node -v returns v0.12.0

@vba-ds
Copy link

vba-ds commented Mar 6, 2015

Have a same problem on FreeBSD 10.1-RELEASE-p5 with nodejs 0.12.0.

#0  0x000000080225e469 in swapcontext () from /lib/libthr.so.3
[New Thread 803407400 (LWP 100767/v8:SweeperThrea)]
[New Thread 803407000 (LWP 100374/v8:SweeperThrea)]
[New Thread 803406c00 (LWP 100331/v8:SweeperThrea)]
[New Thread 803406800 (LWP 100214/OptimizingCompi)]
[New Thread 803406400 (LWP 100081/node)]

(gdb) bt full
#0  0x000000080225e469 in swapcontext () from /lib/libthr.so.3
#1  0x000000080225e062 in sigaction () from /lib/libthr.so.3
#2  <signal handler called>
#3  0x00000008024e244a in _umtx_op () from /lib/libc.so.7
#4  0x00000008024d1c24 in sem_timedwait () from /lib/libc.so.7
#5  0x0000000000b3ce08 in v8::base::Semaphore::Wait ()
#6  0x00000000008c46c4 in v8::internal::SweeperThread::Run ()
#7  0x0000000000b3e52d in v8::base::ThreadEntry ()
#8  0x00000008022594f5 in pthread_create () from /lib/libthr.so.3
#9  0x0000000000000000 in ?? ()

(gdb) t 5
[Switching to thread 5 (Thread 803406400 (LWP 100081/node))]#0  0x000000080259904a in kevent () from /lib/libc.so.7

(gdb) bt full
#0  0x000000080259904a in kevent () from /lib/libc.so.7
#1  0x0000000000b3bad7 in uv__io_poll (loop=0x111b5c0, timeout=<value optimized out>) at ../deps/uv/src/unix/kqueue.c:142
        events = {{ident = 0, filter = 0, flags = 0, fflags = 0, data = 0, udata = 0x0}}
        events = {{ident = 6, filter = -1, flags = 1, fflags = 0, data = 0, udata = 0x0}, {ident = 9, filter = -1, flags = 1, fflags = 0, data = 0, udata = 0x0},


...stripped...

On Ubuntu 14.04 (3.13.0-45-generic) the same code works fine.

@misterdjules misterdjules added this to the 0.12.1 milestone Mar 9, 2015
@misterdjules
Copy link

I haven't had time to reproduce/investigate, but adding it to 0.12.1 to make sure we take care of it asap.

@misterdjules
Copy link

@tjfontaine Assigning to you as discussed during the core team meeting. Please assign to @trevnorris when he can use a FreeBSD 10.1 VM somewhere. Thank you!

@trevnorris
Copy link

Was finally able to reproduce. Though I'm not sure this is a Node thing. Might be a libuv problem?

@trevnorris
Copy link

Here's a simplified test:

require('http').createServer().listen(0, function() {
  process.kill(process.pid, 'SIGINT');
});

Still narrowing down why this is happening.

@trevnorris
Copy link

I've confirmed this is also an issue in io.js. Still investigating why this could possibly happen.

@trevnorris
Copy link

More stack trace info using latest libuv:

#0  0x000000080291f469 in swapcontext () from /lib/libthr.so.3
#1  0x000000080291f062 in sigaction () from /lib/libthr.so.3
#2  <signal handler called>
#3  0x0000000802c79a1a in kill () from /lib/libc.so.7
#4  0x000000000137f149 in uv_kill (pid=41450, signum=2) at ../deps/uv/src/unix/process.c:512
#5  0x00000000012c7dca in node::Kill (args=@0x7fffffffddb0) at ../src/node.cc:1960
#6  0x0000000000b583bc in v8::internal::FunctionCallbackArguments::Call (this=0x7fffffffdeb0, f=0x12c7d30 <node::Kill(v8::FunctionCallbackInfo<v8::Value> const&)>)
    at ../deps/v8/src/arguments.cc:33
#7  0x0000000000ba6a15 in v8::internal::HandleApiCallHelper<false> (args=
        {<v8::internal::Arguments> = {<v8::internal::Embedded> = {<No data fields>}, length_ = 4, arguments_ = 0x7fffffffe0c0}, <No data fields>}, isolate=0x803853000)
    at ../deps/v8/src/builtins.cc:1139
#8  0x0000000000ba64b5 in v8::internal::Builtin_Impl_HandleApiCall (args=
        {<v8::internal::Arguments> = {<v8::internal::Embedded> = {<No data fields>}, length_ = 4, arguments_ = 0x7fffffffe0c0}, <No data fields>}, isolate=0x803853000)
    at ../deps/v8/src/builtins.cc:1156
#9  0x0000000000b9edb0 in v8::internal::Builtin_HandleApiCall (args_length=4, args_object=0x7fffffffe0c0, isolate=0x803853000) at ../deps/v8/src/builtins.cc:1155

@trevnorris
Copy link

I'm going to need more help to continue with this issue.

/cc @joyent/node-coreteam

@indutny
Copy link
Member

indutny commented Mar 19, 2015

Oh yeah, I seen it once. Looking...

@indutny
Copy link
Member

indutny commented Mar 19, 2015

The problem disappears if I'll remove SA_RESETHAND... Looking further

@indutny
Copy link
Member

indutny commented Mar 20, 2015

Looks like sa_handler is messed up somewhere...

@indutny
Copy link
Member

indutny commented Mar 20, 2015

Ha, I think it is a bug in libthr:

#include <assert.h>
#include <pthread.h>
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>

static void handler(int sig) {
  fprintf(stderr, "oh\n");
  fflush(stderr);

  raise(sig);
}


int main() {
  int r;
  struct sigaction sa;

  memset(&sa, 0, sizeof(sa));

  sa.sa_handler = handler;
  sa.sa_flags = SA_RESETHAND;
  sigfillset(&sa.sa_mask);
  r = sigaction(SIGINT, &sa, NULL);
  assert(r == 0);

  kill(getpid(), SIGINT);

  return 0;
}

When compiled without -pthread this program works, when compiled with it - crashes.

@indutny
Copy link
Member

indutny commented Mar 20, 2015

It seems that the problem happens somewhere in thr_sighandler which is supposed to wrap the normal handler, looking into it.

@indutny
Copy link
Member

indutny commented Mar 20, 2015

Just a short follow-up, the stack is in fact:

handle_signal # thr_sig.c:244
thr_sighandler # thr_sig.c:187

Took me awhile to disassemble and find proper chunks, the link for the crash line is:

https://github.com/freebsd/freebsd/blob/master/lib/libthr/thread/thr_sig.c#L244

It seems that thr_sighandler is called with following args: (0x2, 0x10001, 0x7fffffffe640). 0x10001 is what causing the crash. Looking more into it.

@indutny
Copy link
Member

indutny commented Mar 20, 2015

It is odd, because libthr proxies sigact to the libc with a following struct:

{__sigaction_u = {__sa_handler = 0x80082af30, __sa_sigaction = 0x80082af30}, sa_flags = 0x44, sa_mask = {__bits = {0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff}}}

0x44 flags are = SA_SIGINFO | SA_RESETHAND

Going to try passing this without linking to libthr.

@indutny
Copy link
Member

indutny commented Mar 20, 2015

Looks like a kernel bug to me:

#include <assert.h>
#include <pthread.h>
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>

static void handler(int sig, siginfo_t* info, void* u) {
  fprintf(stderr, "oh %d %p %p\n", sig, info, u);
  fflush(stderr);

  raise(sig);
}


int main() {
  int r;
  struct sigaction sa;

  memset(&sa, 0, sizeof(sa));

  sa.sa_sigaction = handler;
  sa.sa_flags = SA_RESETHAND | SA_SIGINFO;
  sigfillset(&sa.sa_mask);
  r = sigaction(SIGINT, &sa, NULL);
  assert(r == 0);

  kill(getpid(), SIGINT);

  return 0;
}

Is producing:

oh 2 0x10001 0x7fffffffe660

/me is looking deeply into the kernel code now.

@indutny
Copy link
Member

indutny commented Mar 20, 2015

It looks like it is using:

Traditional BSD style:
                  void handler(int, int code, struct sigcontext *scp);

For the handler.

@indutny
Copy link
Member

indutny commented Mar 20, 2015

Looks like a combination of SA_SIGINFO and SA_RESETHAND makes kernel skip the SA_SIGINFO.

@indutny
Copy link
Member

indutny commented Mar 20, 2015

Related FreeBSD discussion: https://lists.freebsd.org/pipermail/freebsd-threads/2014-November/005613.html . Looks like a kernel bug for me.

@indutny
Copy link
Member

indutny commented Mar 20, 2015

And the relevant fix: freebsd/freebsd-src@4501dad . I'll ask on a mailing list about possibility of backporting it to 10.x FreeBSD. Looks like we'll need to work around it in io.js.

@indutny
Copy link
Member

indutny commented Mar 20, 2015

And node.js for sure too :)

indutny added a commit to indutny/io.js that referenced this issue Mar 20, 2015
FreeBSD has a nasty bug with SA_RESETHAND reseting the SA_SIGINFO,
that is in turn set for a libthr wrapper. This leads to a crash.
Work around the issue by manually setting SIG_DFL in the signal
handler.

Fix: nodejs/node-v0.x-archive#9326
@indutny
Copy link
Member

indutny commented Mar 20, 2015

Should be fixed by nodejs/node#1218

indutny added a commit to nodejs/node that referenced this issue Mar 20, 2015
FreeBSD has a nasty bug with SA_RESETHAND reseting the SA_SIGINFO,
that is in turn set for a libthr wrapper. This leads to a crash.
Work around the issue by manually setting SIG_DFL in the signal
handler.

Fix: nodejs/node-v0.x-archive#9326
PR-URL: #1218
Reviewed-By: Johan Bergström <bugs@bergstroem.nu>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
@indutny
Copy link
Member

indutny commented Mar 20, 2015

The fix was landed in io.js nodejs/node@b64983d , please consider backporting.

@trevnorris
Copy link

Many thanks @indutny.

/cc @tjfontaine Let's back port this.

misterdjules pushed a commit to misterdjules/node that referenced this issue Mar 28, 2015
This is a backport of b64983d.

Original commit message:

  src: reset signal handler to SIG_DFL on FreeBSD

  FreeBSD has a nasty bug with SA_RESETHAND reseting the SA_SIGINFO,
  that is in turn set for a libthr wrapper. This leads to a crash.
  Work around the issue by manually setting SIG_DFL in the signal
  handler.

  Fix: nodejs#9326
  PR-URL: nodejs/node#1218
  Reviewed-By: Johan Bergström <bugs@bergstroem.nu>
  Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>

Fixes nodejs#9326.
misterdjules pushed a commit to misterdjules/node that referenced this issue Mar 28, 2015
This is a backport of b64983d.

Original commit message:

  src: reset signal handler to SIG_DFL on FreeBSD

  FreeBSD has a nasty bug with SA_RESETHAND reseting the SA_SIGINFO,
  that is in turn set for a libthr wrapper. This leads to a crash.
  Work around the issue by manually setting SIG_DFL in the signal
  handler.

  Fix: nodejs#9326
  PR-URL: nodejs/node#1218
  Reviewed-By: Johan Bergström <bugs@bergstroem.nu>
  Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>

Fixes nodejs#9326.
indutny added a commit that referenced this issue Mar 28, 2015
This is a backport of b64983d.

Original commit message:

  src: reset signal handler to SIG_DFL on FreeBSD

  FreeBSD has a nasty bug with SA_RESETHAND reseting the SA_SIGINFO,
  that is in turn set for a libthr wrapper. This leads to a crash.
  Work around the issue by manually setting SIG_DFL in the signal
  handler.

  Fix: #9326
  PR-URL: nodejs/node#1218
  Reviewed-By: Johan Bergström <bugs@bergstroem.nu>
  Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>

Fixes #9326.

Reviewed-By: Trevor Norris <trev.norris@gmail.com>
PR-URL: #14184
@misterdjules
Copy link

Fixed by 61fe1fe. Thank you @indutny and @trevnorris!

@vba-ds
Copy link

vba-ds commented Apr 6, 2015

Problem with building:

../src/node.cc:2793:34: error: use of undeclared identifier 'nullptr'
  CHECK_EQ(sigaction(signo, &sa, nullptr), 0);
                                 ^
../src/util.h:51:31: note: expanded from macro 'CHECK_EQ'
#define CHECK_EQ(a, b) CHECK((a) == (b))
                              ^
../src/util.h:48:37: note: expanded from macro 'CHECK'
# define CHECK(expression)   assert(expression)
                                    ^
/usr/include/assert.h:54:21: note: expanded from macro 'assert'
#define assert(e)       ((e) ? (void)0 : __assert(__func__, __FILE__, \
                          ^

I guess need to change "nullptr" to "NULL", or just make this include

#include <cstddef>

@misterdjules
Copy link

@vba-ds When integrating the change from io.js, I did not do my job correctly and did not actually test it. My apologies for that and thank you for letting us know!

Since our code base still doesn't enable C++11 features, the change should use NULL and not nullptr. Reopening the issue and adding it to the 0.12.3 milestone.

@misterdjules misterdjules reopened this Apr 6, 2015
@misterdjules misterdjules modified the milestones: 0.12.3, 0.12.2 Apr 6, 2015
misterdjules pushed a commit to misterdjules/node that referenced this issue Apr 14, 2015
61fe1fe backported
b64983d from io.js, but failed to
change nullptr to NULL, which lead to a build break on FreeBSD since the
current build system doesn't enable support for C++11.

This change replaces nullptr by NULL, and has been tested on
FreeBSD 10.1-RELEASE-p8.

Fixes nodejs#9326.
@misterdjules
Copy link

@vba-ds Fixed by #14819.

misterdjules pushed a commit that referenced this issue Apr 15, 2015
61fe1fe backported
b64983d from io.js, but failed to
change nullptr to NULL, which lead to a build break on FreeBSD since the
current build system doesn't enable support for C++11.

This change replaces nullptr by NULL, and has been tested on
FreeBSD 10.1-RELEASE-p8.

Fixes #9326.

Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
PR-URL: #14819
@misterdjules
Copy link

Should be fixed for good in f99eaef. Thank you and sorry for the confusion.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

6 participants