-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Pattern ingesters add a limiter for high eviction rate #13464
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
1 question, otherwise LGTM
} | ||
|
||
func (l *limiter) block() { | ||
l.blockedUntil = time.Now().Add(10 * time.Minute) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What do you think about sampling instead of blocking completely for some time?
We could accept a small %age of the writes to keep building patterns. It might make a nicer experience for the user?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah that's good suggestion. I would like to keep it that way for now until we have all clusters under control and then improve this later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Bare in mind that it will still grow the tree if we do sampling. So I think it's a bit more risky.
In this case it will constantly lock the streams if it keeps evicting.
What this PR does / why we need it:
This automatically throttle streams with high eviction rate (>25% of added sample).
Also reduce pruning ticker and chunk range which allows for more granular pruning but also create smaller chunks for idle patterns.
Checklist
CONTRIBUTING.md
guide (required)feat
PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.docs/sources/setup/upgrade/_index.md
production/helm/loki/Chart.yaml
and updateproduction/helm/loki/CHANGELOG.md
andproduction/helm/loki/README.md
. Example PRdeprecated-config.yaml
anddeleted-config.yaml
files respectively in thetools/deprecated-config-checker
directory. Example PR