-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Client fs streaming API loses data when a file is rotated and a log line is split between two files #11653
Comments
Hi @jstachowiak! Thanks for the detailed report. The log streaming API mostly reuses a lot of the code for streaming files, (ref Were you able to confirm that the logs actually made it to disk? If they don't show up in the streaming API that's a bug for sure but if they don't show up on disk either that's an even bigger bug. |
Hello @tgross, thanks for a quick reply! Yes, I can confirm the logs are saved to disk. |
Hello @tgross! From the code perspective it seems that issue is caused by the incorrect assumption that creation of the log next file means that the stream from previous one is completed. In
As far as I understand, events come from file watcher so there is no way to synchronize them while writing to a file. I did a small test and added a 5 seconds delay in line 812 and then the endpoint worked correctly and @jstachowiak 's test passed. That's definitely not a solution but confirms the findings. I will be happy to discuss and work with you on possible solutions. |
Thanks @arkadiuss, that's helpful! Given that we've received the truncate event here, I'm wondering if it might make sense to add an attempt to read one more frame of data at |
What a synchronization! Thank you for the response. Performing one more read is the solution proposed in above MR. However it's not a truncate event that is received, because nothing is removed from file, only the next one is started. Generally truncate event probably should never happen as far as I correctly understand logging. We received Modified event and cancel and random one is taken. I didn't touch tests yet, will do it. |
Nomad version
Nomad v1.2.2 (78b8c171a211f967a8b297a88a7e844b3543f2b0)
Operating system and Environment details
Ubuntu 20.04.3 LTS 4.19.128-microsoft-standard
Issue
When you stream the contents of a log file using
/client/fs/stream/:alloc_id
or logs with/client/fs/logs/:alloc_id
a data loss occurs. This happens when a log line is being split between two files and the contents are streamed from a file that is currently being rotated. Both files in the file system have the data but the API only sends a frame with the data from the next file, so we lose the first part of the split log line. This makes it very hard to reliably ship logs to a log aggregation system.Files in
alloc/logs
:Captured frames:
In my case, every log line is 204800 bytes (+1 for EOL). Log lines are never batched before being flushed if the frame size has not been hit.
204801 - 40961 (second part of the split log line in "httpd.stdout.1") = 163840
163840 + 819204 (last streamed offset) = 983044 (file size)
Reproduction steps
lineScanLimit = 32 * 1024
.Expected Result
The stream should contain complete logs.
Actual Result
Missing frames.
Job file (if appropriate)
Nomad Server logs (if appropriate)
Nomad Client logs (if appropriate)
The text was updated successfully, but these errors were encountered: