Orphaned kinesis_producer native processes go berserk!

[samcday]

I'm not quite sure how to reproduce this, but I've had it happen on my local laptop for testing and also on a few boxes that I deployed a service to that uses KPL.

Anyway, if you somehow managed to kill the parent KPL process and not the native kinesis_producer binary, it sits around indefinitely and seems to go into a busy-wait spin. It starts chewing up basically a whole CPU doing nothing in particular.

I think it should be more robust than that right? Is it possible to detect that the IPC pipe is dead and just exit gracefully or something?

[samcday]

Okay, I did a bit more digging and found out what's going on.

Background: I have plumbed the KPL (awesome library, btw!) into a Logstash output plugin for writing log events to Kinesis. Anyway, the packaging for Logstash on Ubuntu comes with an init.d script that has a really aggressive shutdown mode. What it will do is wait a few seconds for Logstash to exit gracefully, and if it doesn't, send a SIGKILL to the process.

The SIGKILL immediately kills Logstash, but then orphans the kinesis_producer process KPL spawned. In my particular case, I'm gonna go ahead and disable that behaviour in the Logstash init script since that's awful, but I think this issue is still valid and this is still an issue.

I hate to be that guy, but I found this useful Stack Overflow article I think is relevant. :P Anyway, I think KPL should basically do something like prctl(PR_SET_PDEATHSIG, SIGHUP); and then ensure it exits gracefully on SIGHUP. No idea what to do on Windows / BSD though, but I'm sure you guys are a lot more knowledgable on that sort of stuff than I am ;)

[kevincdeng]

Thanks for reporting this Sam. Are you interfacing with the native KPL directly (i.e. not through the Java wrapper)?

In theory the native KPL should crash from pipe errors, but obviously that's not occurring in your case. Are you able to share your code? That'll make it much easier for us to reproduce.

[Zhichao-Hong]

I have similar problems on OSX. I used the java wrapper. The KPL is called from my javaFX GUI application. When I exit from the GUI (etc, java main exited), sometimes, the kinesis_producer keeps running in the background and keeps the CPU fan busy all the time. Some times, I saw 4 of these running and takes most of the cores away!

[xujiaxj]

Same here. I am using the Java wrapper on Yosemite. "Kill -9" the parent process makes the KPL process go 100% CPU. "CTRL-C" is fine though.

[vmendi]

Same here. We bump into this because we are doing a System.exit(0) at some point in the program, but kinesis_produce process won't die and consumes 100% of the CPU.

[miketehranian]

Same here. I'm using the java KPL in a Tomcat servlet container which is hosting a Spring MVC web service. When running my web service in eclipse on my local machine and subsequently terminating the Tomcat service, kinesis_producer still runs and uses >100% CPU.
The CPU usage spike happens exactly when I terminate my Tomcat instance inside of eclipse.

[kevincdeng]

Hey guys,

We've reproduced this and are working on a fix. Thanks for the bug reports!

[kevincdeng]

Release 0.10.1 is now available and should address this issue.

[samcday]

\0/ Thanks @kevincdeng I'll check it out and get back to you.

[miketehranian]

Confirming that kinesis_producer is being terminated when the parent process terminates. Also, the %CPU usage has gone down in the running state. Thanks for the fix!

[samcday]

Very late reply, but just +1 what @miketehranian said.

We've been using KPL 0.10.1 in production for a few weeks now without issue \0/