stalled outputs will prevent proper shutdown

[colinsurprenant]

This is a meta issue to track related shutdown issues.

---

Problem

Logstash refuses to stop if the output is stalled.

Details

If the output plugins are stalled and not popping events from the @filter_to_output and back pressure propagates, upon shutdown request the LogStash::SHUTDOWN event will never be pushed because queues will be full and so plugins shutdown will not happen.

this relates to shutdown refactor proposals in #3211 and #2477

Related issues:

• exception in output thread silently locks up logstash exception in output thread silently locks up logstash #2130
• outputs can crash silently, locking up entire pipeline outputs can crash silently, locking up entire pipeline #2545
• Logstash stalled/blocked - 1.5.0rc2 logstash stalled/blocked - 1.5.0rc2 #2846
• 1.5.2 fails to shutdown 1.5.2 fails to shutdown #3575
• Service never stop Service never stop #3520
• logstash 1.5.0 freeze logstash 1.5.0 freeze #3440
• Logstash wont kill after rabbit problem Logstash wont kill after rabbit problem #3203

[jsvd]

This is a duplicate of #2130
The trigger condition is an exception instead of stalled output, but the consequence (and fix strategy) is the same.

[colinsurprenant]

#2130 is really focused on the exception handling and this ticket describes the general condition that leads to preventing proper shutdown where "stalled outputs" can be for any number of reason, one of which is that the output plugin crashed. I think we should keep this ticket open because I believe that the proper fix is not limited to improving exception handling...

[purbon]

In #3491 another explanation of this had raised, there are plugins having an infinite retry option, something that is very nice, but in case of having a target system down, it raises this issue as there are no more events being pop of the queues causing shutdown to be impossible.

I add this explanation here as I will close #3491 as is a duplicate of this issue, but knowing infinite retries are also another source of this is important to notice here.

[colinsurprenant]

@purbon ok, perfect. yes, this is exactly the general condition described here where the outputs are stalled. they can be stalled for different reasons like with timeouts or slowdowns communicating with the external destination or stalled on retries.

So the general condition of stalled output plugins need to be dealt with to prevent proper shutdown but also, specific plugins doing infinite retries should be fixed to add a limit on the number of retries.

[purbon]

@colinsurprenant I added this to the redis-output logstash-plugins/logstash-output-redis#9 but I see @jordansissel having concerns with data loss.

[imsaar]

Is there a work around for this problem so get out of this condition before the 1.6 release?

[jsvd]

fixed by #4051:

merged to:
2.1: 1412885
2.x: 79b90b1
master: 5db1d28