Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add multi-line detection support to promtail #399

Closed
wants to merge 1 commit into from

Conversation

davidhao3300
Copy link

Relevant issue: #74

Largely inspired by the code in the fluentd concat plugin: https://github.com/fluent-plugins-nursery/fluent-plugin-concat

Adds a buffer in front of the HTTP client that checks whether a line matches a regex. Lines that do not are merged with a newline with buffered lines. When a new log line matches the regex, the previous lines are flushed, and the new line is buffered. A task that runs every second checks buffered lines and force-flushes them if they're older than a configured timeout.

Regex and timeout are configurable per-scraper-config. By default, the concat functionality is disabled to preserve existing behavior.

TODOs:

  • docs (will do after the overall idea of this PR is okay'd
  • verify that configuration through YAML actually works
  • determine whether we should keep this disabled by default

@CLAassistant
Copy link

CLA assistant check
Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@bryanyork
Copy link

@davidhao3300 Do you plan on continuing here?

@davidhao3300
Copy link
Author

I'd say probably not, but feel free to use any code in this branch!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants