Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mixin: Downgrade severity of forward failures #4064

Merged
merged 1 commit into from
Apr 15, 2021
Merged

Conversation

brancz
Copy link
Member

@brancz brancz commented Apr 15, 2021

We had this alert fire during a rollout, however, it is expected that during rollouts some forward requests fail. As long as ingestion as a whole is still succeeding with a quorum, there's no need to alert on anything.

Signed-off-by: Frederic Branczyk fbranczyk@gmail.com

cc @kakkoyun @squat @metalmatze

kakkoyun
kakkoyun previously approved these changes Apr 15, 2021
Copy link
Member

@kakkoyun kakkoyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. This is a cause-based alert and it makes sense to reduce the severity.

CI is unhappy because it misses generated files. I mark it to auto-merge, it'd be great if you can make the CI happy.

@kakkoyun kakkoyun enabled auto-merge (squash) April 15, 2021 12:10
We had this alert fire during a rollout, however, it is expected that during rollouts some forward requests fail. As long as ingestion as a whole is still succeeding with quorum, there's no need to alert on anything.

Signed-off-by: Frederic Branczyk <fbranczyk@gmail.com>
@kakkoyun kakkoyun merged commit 2a999d2 into main Apr 15, 2021
@kakkoyun kakkoyun deleted the brancz-patch-1 branch April 15, 2021 12:55
someshkoli pushed a commit to someshkoli/thanos that referenced this pull request Apr 21, 2021
We had this alert fire during a rollout, however, it is expected that during rollouts some forward requests fail. As long as ingestion as a whole is still succeeding with quorum, there's no need to alert on anything.

Signed-off-by: Frederic Branczyk <fbranczyk@gmail.com>
Signed-off-by: someshkoli <kolisomesh27@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants