Assertion `(0) == (uv_thread_create(t.get(), start_thread, this))' failed.

[Ashish-Bansal]

Node.js Version:
10.15.0

OS:
Running inside docker(Docker version 18.03.1-ce, build 9ee9f40).
Host OS - Ubuntu 16.04.1 LTS
Docker OS: ubuntu:16.04

Exception:

nodejs run[28]: ../src/node_platform.cc:58:std::unique_ptr<long unsigned int> node::BackgroundTaskRunner::DelayedTaskScheduler::Start(): Assertion `(0) == (uv_thread_create(t.get(), start_thread, this))' failed.
 1: 0x8db900 node::Abort() [nodejs run]
 2: 0x8db9d5  [nodejs run]
 3: 0x9644ea node::BackgroundTaskRunner::BackgroundTaskRunner(int) [nodejs run]
 4: 0x96459b node::NodePlatform::NodePlatform(int, node::tracing::TracingController*) [nodejs run]
 5: 0x8e422b node::Start(int, char**) [nodejs run]
 6: 0x7f5efb235830 __libc_start_main [/lib/x86_64-linux-gnu/libc.so.6]
 7: 0x89ed85  [nodejs run]

  var content = fs.readFileSync(0).toString();

Sometimes nodejs crashes randomly with abort().

[Ashish-Bansal]

Resource related output:

(venv) (issue_js_runner)⚡ % ulimit -a                                                                                                                 
-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             32768
-c: core file size (blocks)         0
-m: resident set size (kbytes)      unlimited
-u: processes                       32768
-n: file descriptors                63536
-l: locked-in-memory size (kbytes)  64
-v: address space (kbytes)          unlimited
-x: file locks                      unlimited
-i: pending signals                 7842
-q: bytes in POSIX msg queues       819200
-e: max nice                        0
-r: max rt priority                 0
-N 15:                              unlimited

(venv) (issue_js_runner)⚡ [1] %cat /proc/sys/vm/max_map_count                                                                                           
600000

(venv) (issue_js_runner)⚡ [1] %cat /proc/sys/kernel/threads-max                                                                                          
16001

[Ashish-Bansal]

If I switch from sync to async stdin read, then sometimes it doesn't print anything and sometimes it prints below given message (when it crashes randomly).

node[5568]: pthread_create: Resource temporarily unavailable

[gireeshpunathil]

@Ashish-Bansal: how many threads / workers are there in your application?

• ulimit -c unlimited, then take a core, launch it and do info thr
  or
  strace -f node <app> 2> out.log if the failure happens pretty fast, then grep 'clone' out.log

let us see how much it is. thanks!

[Ashish-Bansal]

It's quite small application actually. I'm able to reproduce it with just single line i.e. var content = fs.readFileSync(0).toString(); and it would be reading around 10KB from stdin.

Here's the output I got from it when it exits fine.

clone(child_stack=0x7f10c86a8fb0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x7f10c86a99d0, tls=0x7f10c86a9700, child_tidptr=0x7f10c86a99d0) = 31
[pid    30] clone(child_stack=0x7f10c7d13fb0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x7f10c7d149d0, tls=0x7f10c7d14700, child_tidptr=0x7f10c7d149d0) = 32
[pid    30] clone(child_stack=0x7f10c737efb0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x7f10c737f9d0, tls=0x7f10c737f700, child_tidptr=0x7f10c737f9d0) = 33
[pid    30] clone(child_stack=0x7f10c69e9fb0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x7f10c69ea9d0, tls=0x7f10c69ea700, child_tidptr=0x7f10c69ea9d0) = 34
[pid    30] clone(child_stack=0x7f10c6054fb0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x7f10c60559d0, tls=0x7f10c6055700, child_tidptr=0x7f10c60559d0) = 35
[pid    30] clone(child_stack=0x7f10c9b61fb0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x7f10c9b629d0, tls=0x7f10c9b62700, child_tidptr=0x7f10c9b629d0) = 36

---

When ran it under strace and it exited with error: strace: fork: Resource temporarily unavailable

[Ashish-Bansal]

To reproduce this error, I need to run the multiple docker containers(around 4 or 5) running that piece of code. So, yeah I believe it's related to some specific system resource. But which one exactly ? (process/thread limits are quite large).

[Ashish-Bansal]

It was indeed due to limit on no. of processes, being imposed at runtime by a wrapper which was hidden somewhere in the legacy code.

Sorry for the noise!

[gireeshpunathil]

no problem, nice to know that you found the issue, and got it resolved by yourselves!

[waterdrop-chn]

start container with privileged: true

[zhongsheng-chen]

thanks to waterdrop-chn. start container with privileged=true works for me.