monitoring: make some prometheus alert threshold configurable via env.local #187
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
….local Default values are previous hardcoded values. Different organizations with different policies and hardware can now adapt the alert threshold to their specific needs, decreasing false positive alerts. Too much false positive alerts will decrease the importance and usefulness of each alert. Alerts should not feel like spams. Fixes #66.
E2E Test ResultsDACCS-iac Pipeline ResultsBuild URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/593/Result : success BIRDHOUSE_DEPLOY_BRANCH : configurable-alerting-threshold DACCS_CONFIGS_BRANCH : master PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master PAVICS_SDI_BRANCH : master DESTROY_INFRA_ON_EXIT : true PAVICS_HOST : https://host-140-91.rdext.crim.ca PAVICS-e2e-workflow-tests Pipeline ResultsTests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/520/NOTEBOOK TEST RESULTS |
tlvu
force-pushed
the
configurable-alerting-threshold
branch
from
bbe042c
to
190cea0
Compare
E2E Test ResultsDACCS-iac Pipeline ResultsBuild URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/601/Result : success BIRDHOUSE_DEPLOY_BRANCH : configurable-alerting-threshold DACCS_CONFIGS_BRANCH : master PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master PAVICS_SDI_BRANCH : master DESTROY_INFRA_ON_EXIT : true PAVICS_HOST : https://host-140-36.rdext.crim.ca PAVICS-e2e-workflow-tests Pipeline ResultsTests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/534/NOTEBOOK TEST RESULTS |
E2E Test ResultsDACCS-iac Pipeline ResultsBuild URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/602/Result : success BIRDHOUSE_DEPLOY_BRANCH : configurable-alerting-threshold DACCS_CONFIGS_BRANCH : master PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master PAVICS_SDI_BRANCH : master DESTROY_INFRA_ON_EXIT : true PAVICS_HOST : https://host-140-36.rdext.crim.ca PAVICS-e2e-workflow-tests Pipeline ResultsTests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/535/NOTEBOOK TEST RESULTS |
E2E Test ResultsDACCS-iac Pipeline ResultsBuild URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/603/Result : success BIRDHOUSE_DEPLOY_BRANCH : configurable-alerting-threshold DACCS_CONFIGS_BRANCH : master PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master PAVICS_SDI_BRANCH : master DESTROY_INFRA_ON_EXIT : true PAVICS_HOST : https://host-140-36.rdext.crim.ca PAVICS-e2e-workflow-tests Pipeline ResultsTests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/536/NOTEBOOK TEST RESULTS |
matprov
reviewed
Aug 31, 2021
Comment on lines
22
to
23
| export PROMETHEUS_HostUnusualDiskReadLatency_ALERT=100 # milli seconds | ||
| export PROMETHEUS_HostUnusualDiskWriteLatency_ALERT=100 # milli seconds |
matprov
reviewed
Aug 31, 2021
| for: 5m | ||
| labels: | ||
| severity: warning | ||
| annotations: | ||
| summary: "Host swap is filling up (instance {{ $labels.instance }})" | ||
| description: "Swap is filling up (>80%)\n VALUE = {{ $value }}\n LABELS: {{ $labels }}" | ||
| description: "Swap is filling up (> $PROMETHEUS_HostSwapIsFillingUp_ALERT %)\n VALUE = {{ $value }}\n LABELS: {{ $labels }}" |
There was a problem hiding this comment.
unnecessary space before the %
There was a problem hiding this comment.
I inserted the space on purpose so I do not have to surround PROMETHEUS_HostSwapIsFillingUp_ALERT with {}. Same reason for all the extra spaces inserted elsewhere.
matprov
approved these changes
Aug 31, 2021
github-actions
bot
added
ci/operations
E2E Test ResultsDACCS-iac Pipeline ResultsBuild URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/604/Result : success BIRDHOUSE_DEPLOY_BRANCH : configurable-alerting-threshold DACCS_CONFIGS_BRANCH : master PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master PAVICS_SDI_BRANCH : master DESTROY_INFRA_ON_EXIT : true PAVICS_HOST : https://host-140-91.rdext.crim.ca PAVICS-e2e-workflow-tests Pipeline ResultsTests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/537/NOTEBOOK TEST RESULTS |
E2E Test ResultsDACCS-iac Pipeline ResultsBuild URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/605/Result : failure BIRDHOUSE_DEPLOY_BRANCH : configurable-alerting-threshold DACCS_CONFIGS_BRANCH : master PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master PAVICS_SDI_BRANCH : master DESTROY_INFRA_ON_EXIT : true PAVICS_HOST : https:// PAVICS-e2e-workflow-tests Pipeline ResultsTests URL :NOTEBOOK TEST RESULTS
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Default values are previous hardcoded values so this is fully backward compatible
Different organizations with different policies and hardware can now
adapt the alert threshold to their specific needs, decreasing false
positive alerts.
Too much false positive alerts will decrease the importance and
usefulness of each alert. Alerts should not feel like spams.
Not all alerts are changed to make configurable. I've only changed those that I think are most likely to need customization or that logically should be configurable.
Fixes #66.