promtail: Allow docker_sd_config to handle log lines longer than 64k #6256
Conversation
In promtail's docker_sd_config, parsing of a container's logs would fail if any log line was over 64k bytes in length. This is because bufio.Scanner uses a 64k buffer by default. Since there doesn't appear to be a way to elegantly catch `bufio.ErrTooLong` and carry on, the parsing approach was changed to bufio.Reader which can handle lines of arbitrary length.
|
./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell. + ingester 0%
+ distributor 0%
+ querier 0%
+ querier/queryrange 0%
+ iter 0%
+ storage 0%
+ chunkenc 0%
+ logql 0%
+ loki 0% |
| // https://devmarkpro.com/working-big-files-golang | ||
| func readLine(r *bufio.Reader) (string, error) { | ||
| var ( | ||
| isPrefix = true | ||
| err error | ||
| line, ln []byte | ||
| ) | ||
|
|
||
| for isPrefix && err == nil { | ||
| line, isPrefix, err = r.ReadLine() | ||
| ln = append(ln, line...) | ||
| } | ||
|
|
||
| return string(ln), err | ||
| } |
There was a problem hiding this comment.
Do you mind putting this function into a utility package, e.g. clients/pkg/promtail/common? I think it may be used in various places.
There was a problem hiding this comment.
I'm fine either way on that actually, we can also wait until we need it somewhere else to figure out the best package for it to go into, or what the interface should look like, ie. there might be more duplication to reduce if we refactor our the section in process for processing a whole file?
There was a problem hiding this comment.
Yeah I'm fine either way, happy to move it if that's the desired approach...
| // https://devmarkpro.com/working-big-files-golang | ||
| func readLine(r *bufio.Reader) (string, error) { | ||
| var ( | ||
| isPrefix = true | ||
| err error | ||
| line, ln []byte | ||
| ) | ||
|
|
||
| for isPrefix && err == nil { | ||
| line, isPrefix, err = r.ReadLine() | ||
| ln = append(ln, line...) | ||
| } | ||
|
|
||
| return string(ln), err | ||
| } |
There was a problem hiding this comment.
I'm fine either way on that actually, we can also wait until we need it somewhere else to figure out the best package for it to go into, or what the interface should look like, ie. there might be more duplication to reduce if we refactor our the section in process for processing a whole file?
| @@ -89,7 +89,7 @@ func Test_TargetManager(t *testing.T) { | |||
| require.True(t, ta.Ready()) | |||
|
|
|||
| require.Eventually(t, func() bool { | |||
| return len(entryHandler.Received()) >= 5 | |||
| return len(entryHandler.Received()) >= 6 | |||
There was a problem hiding this comment.
what caused this value to change? I don't see any changes to the test setup.
There was a problem hiding this comment.
I inserted a Very Long Log Line into flog.log at line 6 to test against (over 65k). I had to increase by one here to read in that extra line.
There was a problem hiding this comment.
got it, and that's a binary file which is why I'm not seeing the diff, thanks!
What this PR does / why we need it:
In promtail's docker_sd_config, parsing of a container's
logs would fail if any log line was over 64k bytes in length.
This is because bufio.Scanner uses a 64k buffer by default.
Since there doesn't appear to be a way to elegantly catch
bufio.ErrTooLongand carry on, the parsing approach was changedto bufio.Reader which can handle lines of arbitrary length.
Which issue(s) this PR fixes:
Fixes #6257
Special notes for your reviewer:
Originally I considered increasing the buffer or allowing for a config parameter to do the same, but that would only raise the ceiling of parsable log line lengths. Since there doesn't seem to be a way to catch
bufio.ErrTooLongwhen usingScanner, it seemed better to just go with theReaderapproach and handle lines of arbitrary length.Log lines of this length can be problematic by nature and it may be prudent to add a configuration to truncate parsed log lines to a certain maximum length, but I think that concern is separate enough to avoid conflating it here.
Checklist
CHANGELOG.md.docs/sources/upgrading/_index.md