Make Logparser Plugin Check For New Files

[njwhite]

Required for all PRs:

• CHANGELOG.md updated (we recommend not updating this until the PR has been approved by a maintainer)
• Sign CLA (if not already signed)
• README.md updated (if adding a new plugin)

Check in the Gather metric to see if any new files matching the glob
have appeared. If so, start tailing them from the beginning.

[sparrc]

Please put a note in the sample config that new files will always be tailed from the beginning.

[njwhite]

@sparrc thanks - I've updated the PR with your review feedback

[sparrc]

looks good, please update the changelog!

[sparrc]

I believe this closes #1829

[njwhite]

@sparrc thanks - I've updated the changelog.

[kiplandiles]

Telegraf v1.2.1 (git: release-1.2 3b6ffb3)
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=12.04
DISTRIB_CODENAME=precise
DISTRIB_DESCRIPTION="Ubuntu 12.04.5 LTS"

Was this fixed? I just fired up a logParser input for a haproxy.log and noticed that Grafana quit displaying the metrics from the haproxy.log. When I checked telegraf.log I see the information below. At 16:13:31 I started telegraf. At 16:17:01 it looks like telegraf is attempting to handle the log rotation and at 16.17.02 it looks to have reopened the new haproxy.log file. That's when I lost my metrics in Grafana. I "reloaded" telegraf (service reload telegraf) at 16:34 and started collecting metrics again. Maybe telegraf need to HUP itself as part of this process.

2017/04/19 16:13:31 Seeked /var/log/haproxy.log - &{Offset:0 Whence:2}
2017/04/19 16:17:01 Re-opening moved/deleted file /var/log/haproxy.log ...
2017/04/19 16:17:01 Successfully reopened /var/log/haproxy.log
2017/04/19 16:17:02 Re-opening moved/deleted file /var/log/haproxy.log ...
2017/04/19 16:17:02 Successfully reopened /var/log/haproxy.log
2017/04/19 16:34:09 Seeked /var/log/haproxy.log - &{Offset:0 Whence:2}

Relevant portion of my telegraf configuration

[global_tags]
  role = "haproxy"
[[inputs.logparser]]
  ## file(s) to tail:
  files = ["/var/log/haproxy.log"]
  from_beginning = false
  name_override = "haproxy_log"
  ## For parsing logstash-style "grok" patterns:
  [inputs.logparser.grok]
    patterns = ["%{HAPROXYHTTP}"]
    custom_patterns = '''
        HAPROXYTIME %{HOUR}:%{MINUTE}:%{SECOND}
        HAPROXYDATE %{MONTH:haproxy_month} %{MONTHDAY:haproxy_day} %{HAPROXYTIME:haproxy_time}
        HAPROXYHTTP %{HAPROXYDATE} %{IPORHOST:syslog_server} %{SYSLOGPROG}: %{IP:client_ip} %{WORD:method} %{URIPATHPARAM:http_request} HTTP/%{NUMBER} %{NUMBER:status:int} %{NUMBER:request_bytes:int} %{INT}
    '''

[danielnelson]

@kiplandiles See #1829

[kiplandiles]

@danielnelson - Thanks. I actually started at #1829 and it referred me here. I'm a bit of a GitHub newbie so sorry if I should have posted there instead. So do I understand that this won't be fixed until 1.3? @sparrc commented that this closed 1829.

[danielnelson]

I reopened that issue because there are still problems. I believe it is caused by a bug in the upstream project. Perhaps this issue: hpcloud/tail#97

I'm not sure when it will get fixed, but I don't consider it a blocker because it isn't a regression. Perhaps there is already a solution floating around that we could merge into our fork https://github.com/influxdata/tail.

The next release of Telegraf will be 1.3, not planning any bug fix releases before then.

[kiplandiles]

Thanks again. Guess I will handle this issue myself with logrotate.

postrotate
	reload telegraf >/dev/null 2>&1 || true
endscript

We shall see how that works out. Already have to add telegraf to the adm group just to access the haproxy log file so logParser seems to require a bit of tweaking.
usermod -aG adm telegraf

[d0o0]

how to closed the fd of the deleted files? since tail only execute closefile when telegraf exits