-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Store: fix error handling on limits #6171
Store: fix error handling on limits #6171
Conversation
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice find
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
As always, I forget to goimports my files 😭 Fixed in 60cf9a0. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, do we need to fix this in other stores?
@fpetkovski from what I checked just now, yes. I can follow up with a PR fixing things around the |
Changes
I noticed from running Thanos Store in production that limits were being hit quite often and this triggered an alert based on percentage of failed requests. A deep investigation allowed me to identify that the grpc codes for requests hitting limits made no sense: they were mostly "Unknown" and "Aborted". This motivated me to fix such logic.
ResourceExhausted
.Verification
E2E tests changed and passing.