Not able to read rotated log file without missing lines

[kingluo]

Feature Request

the rotate procedure:

• step1: the program keeps appending log entries into the log file
• step2: mv logfile logfile.1
• step3: kill -USR1 program
• step4: the program reopens the log file

Current behavior:

If I specify one file in conf:

files = ["logfile"]

telegraf losts log entries written between step2 and step4.

If I specify glob in the file:

files = ["logfile*"]

Then telegraf would resend all log entries in logfile.1 (mostly duplicated from logfile) from the beginning, because logfile.1 is created when telegraf running.

Both ways give wrong result.

Desired behavior:

telegraf should not stop reading the old renamed file, because the program may keep writing into that file for a while. The watch period could be configured.

[sgtsquiggs]

This issue is causing me problems right now - when telegraf is sent a SIGHUP to reload its configuration, it will miss log lines written during the reload. I'm not sure how to solve this elegantly since reloading destroys and reloads all plugin instances. Perhaps tail could hold a global state to keep track of offsets of files watched for the duration of the telegraf process.

[danielnelson]

I think this would be okay, sure there are some corner cases where it doesn't work as expected such as more than one plugin tailing the same file or if the file is switched out exactly at reload time, but I think overall this would work.

[danielnelson]

Reopening since #6074 won't actually fix the original issue here.

[rajat23495]

any update on this, it seems many log lines are missing which is resulting in loss of some significant data points.

[reimda]

It sounds this caused by using the tail plugin with from_beginning=false. In step 4, telegraf opens the new file, seeks to the end and skips any data that was written when it was restarting, then waits for new data. I don't see a way to change this behavior without the tail plugin saving some state. It needs to know that the file it was reading has been moved. Then if it has, it needs to start reading from the beginning of the file.

Since plugins don't have a way to save state, it's not currently possible to do this.

There is a PR that adds plugin state (#9476). Once that has been finished, reviewed, and merged, we can look into modifying the tail plugin to use it, specifically to keep track of whether the file has been moved. I suspect we'll have to have it record the inode of the file since we're dealing with renaming files.

[srebhan]

@kingluo and @rajat23495 I cannot reproduce this with the latest version of Telegraf. One potential source of trouble is if you use copytruncate-style logrotate as Telegraf might not have finished reading the last lines before the file is trucated. This can be avoided by using the "move" style as Telegraf will still have a reference to the old file-descriptor...

So my request is to retest with the latest version of Telegraf and let me know if that works. If it doesn't, please provide a setup to reliably reproduce the issue.

[telegraf-tiger]

Hello! I am closing this issue due to inactivity. I hope you were able to resolve your problem, if not please try posting this question in our Community Slack or Community Forums or provide additional details in this issue and reqeust that it be re-opened. Thank you!