Pipeline shutdown can be blocked by stdin input

[jordansissel]

For whatever reason, I can't figure out how to close stdin in JRuby and interrupt any currently-blocked reads on stdin.

This confuses users and is generally annoying.

The symptom is that ^C should terminate logstash cleanly, but stdin never really stops so the pipeline stays up until you close stdin as a user (via ^D)

(related: #1767)

[jordansissel]

From another thread, I have attempted:

• java.lang.System.in.close()
• $stdin.close
• Thread#raise
• IO::select([$stdin], ...)

None are effective. Closing doesn't interrupt any active reads. IO::select always says $stdin is ready to read even when there is no data.

[jordansissel]

Contrast to MRI 2.1.2:

• $stdin.sysread(1) - gets a RuntimeError: "`sysread': unhandled exception"
• IO.select([$stdin], nil, nil, nil) - gets same error as above, except it says `select'.

[jordansissel]

rvm default do ruby -e 't = Thread.new { begin ; p :sysread => $stdin.sysread(1); puts "Done"; rescue => e; p :e => e; raise; end }; sleep 3; puts "Closing $stdin"; $stdin.close; puts "Interrupting "; t.raise; t.join'

[talevy]

looks related: http://jira.codehaus.org/browse/JRUBY-5723

[colinsurprenant]

Just a few notes, as I looked at this a bit last evening.

• stdin but stdio in general is not selectable in Java NIO. (Not sure if it is in NIO2). Doing IO.select([$stdin], nil, nil, 0) in JRuby will not work as intended.
• blocking read on stdin in a thread in Java NIO is non-interruptible, period. ❗

The only options I see for this are

• if NIO2 support selectable stdio we could implement this using NIO2, but only for Java 7+
• implement our own stdin reader either by just wrapping needed POSIX system calls with FFI and maybe injecting a little C for better performance.

[jordansissel]

(reposting from hipchat)
+1 on FFI because it will work on both MRI and JRuby.

[seanfahey]

Is it possible to have the Getting Started documentation updated, until the issue is fixed? http://logstash.net/docs/1.4.2/tutorials/getting-started-with-logstash in the "Logstash in two commands" section.

[josegonzalez]

EDIT: Seems my issue is actually #1779. Sorry!

Note: I'm unable to get it to close, even with CTRL+D. I have to manually kill the agent using kill -term or kill -9. I'm running under the openjre in case that helps.

Also, here is the config i'm running with, which doesn't use stdio as far as I can tell:

/opt/logstash/bin/logstash -e 'input { redis { key => "metricsd" data_type => "list" } } output { elasticsearch { host => localhost } stdout { codec => "json" } }'

When I had it running under stdin instead of redis, I was able to kill the agent, though it did take a while for it to die.

I can also provide a Vagrantfile where this issue is reproducible (at least for me) if that helps.

[colinsurprenant]

@josegonzalez for the shutdown + redis issue see #2871

[colinsurprenant]

just a quick update to say I found a way to extract an InterruptibleChannel from System.in - I'll be making a test integration in the stdin input plugin which should allow a simple ^C to correctly shutdown logstash when using stdin input.

I looks like this solution only works on Java 8 though.

[jordansissel]

@colinsurprenant I'm OK with this - Java 7 is EOL anyway, so I think it's reasonable to expect that Java 8 should become more common very soon.

[jordansissel]

<3 you so much for finding this solution

[josegonzalez]

Yay java 8! Good work @colinsurprenant :)

[colinsurprenant]

landed the jruby-stdin-channel gem for it. just need to insert that into the stdin input and see how that behave :P

[colinsurprenant]

inserted in the stdin input in logstash-plugins/logstash-input-stdin#3

[jsvd]

please follow this in logstash-plugins/logstash-input-stdin#3