Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add alerts for tracelistener database errors #795

Open
1 task
Pitasi opened this issue May 19, 2022 · 0 comments
Open
1 task

Add alerts for tracelistener database errors #795

Pitasi opened this issue May 19, 2022 · 0 comments

Comments

@Pitasi
Copy link
Contributor

Pitasi commented May 19, 2022

Use this query as a base:

{container="trace-store"} |= "database error" != "affected rows are zero"

(we currently log quite a log of "affected rows are zero" but we can ignore them, in the future we won't have these logs).

These kind of errors are really bad. They mean we couldn't write to database so we have an inconsistent state, usually manifested as wrong balances for users. We should be notified early instead of waiting for users to complain.

To recover from this state we need a bulk import (and maybe to fix a bug in the Tracelistener).

DoD:

  • Add a new alert in Alert Manager for each of these errors, use slack as a notification channel
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant