Systemd does not see all shutdowns as failures

[djsly]

Fixes: #2717

[root@<hostname> ~]# systemctl status telegraf -l
● telegraf.service - The plugin-driven server agent for reporting metrics into InfluxDB
   Loaded: loaded (/usr/lib/systemd/system/telegraf.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Tue 2017-04-25 10:33:40 EDT; 14min ago
     Docs: https://github.com/influxdata/telegraf
 Main PID: 7857 (code=killed, signal=PIPE)

This occurs when we restart systemd-journald, which cause a crash/stop of telegraph seen as a STDPIPE

See: https://bugs.freedesktop.org/show_bug.cgi?id=84923#c9
With affected services like Consul: hashicorp/consul#1688
and Prometheus: https://github.com/voxpupuli/puppet-prometheus/pull/3/files

Required for all PRs:

• CHANGELOG.md updated (we recommend not updating this until the PR has been approved by a maintainer)
• Sign CLA (if not already signed)

[danielnelson]

Would it make sense to use RestartForceExitStatus=SIGPIPE to reduce side effects?

[djsly]

I could test to ensure it is doing the same work.
Note that I found those issues under influxData

influxdata/docs.influxdata.com-ARCHIVE#548
where influxDB / telegraf / kapacitor were reported to have the issue.

I can't seem to find the DOC changes though... And the provided fix is to put always
@rossmcdonald do you know where I can find the DOC referencing this issue ?

[djsly]

@danielnelson I have tested with RestartForceExitStatus=SIGPIPE it restart telegraf as well.
Update the PR

[danielnelson]

So, it looks like no code change was made in influxdb or kapacitor, and in influxdata/influxdb#7040 there was some concern that this option is no available on older systemd versions. Perhaps enough time has passed that this is no longer a concern?

@rkuchan Do you know if the documentation change happened?

[djsly]

I tested on CentOS 7.3 and the version of systemd is still problematic (centOS 7.1 is running the same version)

systemctl --version
systemd 219

I understand that the latest verison of systemd might be fixing the issue but until the latest releases of Os distribution start using it, it could be a pain to have to manage the unit file for each of the services...

[rkuchan]

@danielnelson - I can't find the doc change anywhere. Want me to reopen the issue?

[danielnelson]

@rkuchan Yeah

[danielnelson]

According to influxdata/influxdb#7040 (comment), the RestartForceExitStatus option was added in systemd 215. Do we know what happens on versions before 215 if we use this option?

[djsly]

@danielnelson: Unfortunately, I don't have a system with systemd --version 215 to test this on.

[danielnelson]

Looks like Debian jessie has 215, but SLES 12 has only 210. We should fire up a VM and check if adding the option causes any issues, if it is okay I think we can merge this.

@rkuchan Actually lets wait and see how things go on SLES before adding to the docs.

[danielnelson]

SLES 12 SP2 actually has systemd-228-117.12.x86_64, so I think we are good.

@djsly Can you add this to the 1.4 changelog? I will probably port this over to the 1.3 branch after 1.3.0 is released.

No change will be needed in the docs since this should solve the problem for all of our targets.

[djsly]

@danielnelson let me know if this is good. Thanks

[danielnelson]

Can you put the changelog entry in the 1.4 section, I'll move it to 1.3.x later if it goes in. Looks like you need to rebase to get rid of all the unrelated commits.

[djsly]

@danielnelson Done. Sorry I guess when I pull I didnt get the latest CHANGELOG.
Everything should be good now.

[phemmer]

Don't we want one or the other here, not both?
With Restart=always, RestartForceExitStatus doesn't have any effect.

[djsly]

Fixed, sorry , got confused doing merge conflict while multi tasking . thanks for catching this up!