Include zeros for missing queue states in Oban polling metrics #245
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
DB query will only count jobs that exist. When counts for a particular queue state combination disappears from one poll to the next, some tools will show the stale last-seen count, when the true count is zero. This change will include zeros for all missing queue + state combinations. Queues are taken from the Oban config, and Oban exposes all possible states in `Oban.Job.states/0`. This addresses issue akoutmos#202
|
This is a real issue we are running into as well. Dashboards get stuck on high counts and trigger alerts because they are unable to go to 0. |
|
This fix only works in the classic oban setup. If you use https://hexdocs.pm/oban/2.11.0/dynamic_queues.html Something like this should handle both cases: EDIT: |
|
Taking a look at this right now @martinsvalin-kivra & @benonymus. Should be able to get this merged in by EOD and a cut a new release. Thanks! |
|
Thank you! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Change description
This changes Oban polling metrics to include zeros for all missing queue + state combinations in the Oban jobs table. Queues are taken from the Oban config, and Oban exposes all possible states in
Oban.Job.states/0.What problem does this solve?
The polling query will only count jobs that exist in the table, so naturally no zeros will be returned. When counts for a particular queue state combination is present and then disappears from one poll to the next, some tools will show the stale last-seen count, when the true count is zero. A spike in available or executing jobs that is quickly processed can lead to a misleading count hanging around.
My team at work had this issue, and we're running a copy of the Oban plugin for now, with these changes.
A possible downside of this change is a larger metrics payload, in cases with a lot of mostly-empty queues. We only have 5 fairly busy queues.
Issue number: #202
Checklist
Sorry about the absence of tests. It looked like adding tests for this would be fairly invasive in the current test suite.