-
Notifications
You must be signed in to change notification settings - Fork 24
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add FLP-based deduper options #591
base: main
Are you sure you want to change the base?
Conversation
Skipping CI for Draft Pull Request. |
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
const ( | ||
FLPDeduperDisabled FLPDeduperMode = "Disabled" | ||
FLPDeduperDrop FLPDeduperMode = "Drop" | ||
FLPDeduperSample FLPDeduperMode = "Sample" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jpinsonneau a possibility could be to add a "Merge" mode here that would involve infinispan like in your PoC
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sounds good; but should we offer all of these or just support one or two modes in the end ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see pros and cons on every mode and no clear "winner"
- Disabled to make sure we get every flow / not loosing anything
- Infinispan-based is similar but with performance impacts (positive on Loki, negative on FLP) and adds a new component, so it requires more configuration
- Drop is the best for overall performance but looses data
- Sample offers a compromise between Drop and Disabled, providing statistical samples of dropped flows
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #591 +/- ##
==========================================
- Coverage 67.38% 66.63% -0.75%
==========================================
Files 65 65
Lines 7987 8081 +94
==========================================
+ Hits 5382 5385 +3
- Misses 2276 2365 +89
- Partials 329 331 +2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
PR needs rebase. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Description
FLP-based dedup allows to decrease Loki CPU / memory / storage a lot (~50%) at the cost of minimal loss in data accuracy (e.g. loosing interfaces involved in egress traffic)
Dependencies
n/a
Checklist
If you are not familiar with our processes or don't know what to answer in the list below, let us know in a comment: the maintainers will take care of that.