nomad logs: output is incomplete and disordered

[fho]

Nomad version

reproduced it with 0.8 and 0.7.1

Issue

Running nomad logs multiple times for an allocation that wrote more then X KB of output
to STDOUT produces different results (even when the output isn't changing anymore).
Lines are missing and/or the output is disordered (line X+1 appears before line X).

This seems only to happen when the output of an allocation is bigger then a certain size (1004K of output is sufficient to reproduce it).

I expect that when an allocation is not printing anything new to STDOUT/STDERR that:

• the output from nomad logs (starting from the offset) is exactly the same then alloc/logs/*.stdout.0, no missing lines, lines appear at the same offset,
• running nomad logs multiple times always outputs the same

Reproduction steps

• run a nomad job that produces >= 1004K of STDOUT output (less output may also be sufficient) and then sleep forever
• run nomad logs multiple times, compare the output

---

• bash script to generate output: https://gist.github.com/fho/c35d02531b9d38ea4fce273c9b5872a6#file-produce-logs-sh
• nomad job file to run it: https://gist.github.com/fho/c35d02531b9d38ea4fce273c9b5872a6#file-produce-logs-sh
• wait until the job is started and it produced the output
• run the following to collect output of multiple nomad logs runs:

mkdir -p /tmp/logs
for ((i=0;i<30;i++)); do nomad logs <ALLOC_ID> | tee /tmp/logs/$i; done

• compare them, e.g. with md5sum and diff, most times not even 2 runs of the command produced the same output

[schmichael]

Thanks for the thorough report @fho! I was easily able to reproduce the bug. I ran log 10 times and it produced 3 distinct results. We'll look into it as soon as possible.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.