Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Monitoring: add alert rules and alert handling (deduplicate, group, route, silence, inhibit) #59

Merged
merged 29 commits into from
Jul 11, 2020

Conversation

tlvu
Copy link
Collaborator

@tlvu tlvu commented Jul 8, 2020

This is a follow up to the previous PR #56 that added the monitoring itself.

Ping @dbyrns @matprov you might be interested in this.

Added cAdvisor and Node-exporter collection of alert rules found here https://awesome-prometheus-alerts.grep.to/rules with a few fixing because of errors in the rules and tweaking to reduce false positive alarms (see list of commits). Great collection of sample of ready-made rules to hit the ground running and learn PromML query language on the way.

2020-07-08-090953_474x1490_scrot

Added Alertmanager to handle the alerts (deduplicate, group, route, silence, inhibit). Currently the only notification route configured is email but Alertmanager is able to route alerts to Slack and any generic services accepting webhooks.

2020-07-08-091150_1099x669_scrot

2020-07-08-091302_1102x1122_scrot

This is an initial attempt at alerting. There are several ways to tweak the system without changing the code:

  • To add more Prometheus alert rules, volume-mount more *.rules files to the prometheus container.
  • To disable existing Prometheus alert rules, add more Alertmanager inhibition rules using ALERTMANAGER_EXTRA_INHIBITION via env.local file.
  • Other possible Alertmanager configs via env.local: ALERTMANAGER_EXTRA_GLOBAL, ALERTMANAGER_EXTRA_ROUTES, ALERTMANAGER_EXTRA_RECEIVERS.

What more could be done after this initial attempt:

  • Possibly add more graphs to Grafana dashboard since we have more alerts on metrics that we do not have matching Grafana graph. Graphs are useful for historical trends and correlation with other metrics, so not required if we do not need trends and correlation.

  • Only basic metrics are being collected currently. We could collect more useful metrics like SMART status and alert when a disk is failing.

  • The autodeploy mechanism can hook into this monitoring system to report pass/fail status and execution duration, with alerting for problems. Then we can also correlate any CPU, memory, disk I/O spike, when the autodeploy runs and have a trace of previous autodeploy executions.

I had to test these alerts directly in prod to tweak for less false positive alert and to debug not working rules to ensure they work on prod so these changes are already in prod ! This also test the SMTP server on the network.

See rules on Prometheus side: http://pavics.ouranos.ca:9090/rules, http://medus.ouranos.ca:9090/rules

Manage alerts on Alertmanager side: http://pavics.ouranos.ca:9093/#/alerts, http://medus.ouranos.ca:9093/#/alerts

Part of issue #12

tlvu added 26 commits July 3, 2020 14:34
Basic AlertManager config is provided with extensions via env.local env var.

Prometheus is configured to talk to Alertmanager.

Alerting rules in Prometheus will be for another commit.
Found a bunch of ready made alert rules, disabled those that did not
seems to parse properly.

The rest did not assess if they work correctly.
…oint filter was wrong anyways, should have been mountpoint="/"
…t was wrong anyways, should have been mountpoint="/"
…tency rate range, probably due to our scrape_interval of 60s
… and node-exporter are local since same hostname
…bleErrorsDetected, HostNetworkReceiveErrors, HostNetworkTransmitErrors

Yml syntax error, need to espace double-quote in {{ printf \"%.0f\" $value }}.
…r so will never exceed threshold so will never fire
When memory limit not set (added container_spec_memory_limit_bytes > 0)
and when not a container (missing "name" label).
…mporary containers spawned for the auto deployment
… new threshold from 1 day observation on prod, might need tweaking again later
@tlvu tlvu requested a review from tlogan2000 July 8, 2020 14:47
tlvu added 2 commits July 10, 2020 23:35
Switching from .md format to .rst format for automated table of content
generation.
To have all components documentation at the same place.
@tlvu tlvu merged commit 582dd9c into master Jul 11, 2020
@tlvu tlvu deleted the add-alert-handling branch July 11, 2020 03:50
@tlvu
Copy link
Collaborator Author

tlvu commented Jul 11, 2020

Tagged 1.10.1.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants