Collapse multiline logs based on a start line. #3024
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Summary: This is a very simple approach based on grafana#1380 to provide multiline or block log entries in promtail. A `multiline` stage is added to pipelines. This stages matches a start line. Once a start line is matched all following lines are appended to an entry and not passed on to downstream stages. Once a new start line is matched the former block of multilines is sent. If now new line arrives withing `max_wait_time` the block is flushed to the next stage and a new block is started.
cyriltovena
reviewed
Dec 3, 2020
Codecov Report
@@ Coverage Diff @@
## master #3024 +/- ##
==========================================
- Coverage 62.63% 62.59% -0.04%
==========================================
Files 185 186 +1
Lines 15731 15822 +91
==========================================
+ Hits 9853 9904 +51
- Misses 4956 4992 +36
- Partials 922 926 +4
|
cyriltovena
reviewed
Dec 3, 2020
cyriltovena
reviewed
Dec 3, 2020
cyriltovena
reviewed
Dec 3, 2020
cyriltovena
reviewed
Dec 3, 2020
Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>
cyriltovena
reviewed
Dec 3, 2020
cyriltovena
reviewed
Dec 3, 2020
cyriltovena
approved these changes
Dec 3, 2020
There was a problem hiding this comment.
LGTM Great work @jeschkies
I know some folks that will be happy !
There's one last comment before we merge, about logging.
EDIT: and the mess I made with the func :p
|
Wow, great work @jeschkies 🥇 |
owen-d
approved these changes
Dec 3, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
This is a very simple approach based on #1380 to provide multiline
or block log entries in promtail.
A
multilinestage is added to pipelines. This stages matches a startline. Once a start line is matched all following lines are appended
to an entry and not passed on to downstream stages. Once a new start
line is matched the former block of multilines is sent.
If no new line arrives within
max_wait_timethe block is flushed tothe next stage and a new block is started.
If a block exceeds
max_linesit is flushed as well.Which issue(s) this PR fixes:
Resolves #74
Checklist