Transition basket tasks to use rq #1069
Conversation
robhudson
force-pushed
the
rq-queues
branch
3 times, most recently
from
084fe06 to
94b2371
Compare
Codecov Report
@@ Coverage Diff @@
## main #1069 +/- ##
==========================================
+ Coverage 84.18% 85.29% +1.11%
==========================================
Files 26 26
Lines 2491 2231 -260
==========================================
- Hits 2097 1903 -194
+ Misses 394 328 -66
... and 3 files with indirect coverage changes 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
|
|
||
| if job.is_failed: | ||
| statsd.incr(f"{task_name}.retry_max") | ||
| statsd.incr("news.tasks.retry_max_total") |
There was a problem hiding this comment.
I'm probably being a bit dim, but am failing to visualise what the resulting data looks like for this situation - how does retry_max_total fit with failed jobs -- I get that one MAX_RETRIES is hit the job fails, so is this stat tracking the number of times that exhaustion happens?
There was a problem hiding this comment.
Yeah, we'd get a count of how many times a specific task hits the max retries, as well as how many times all jobs hit the max retries.
|
|
||
| if ignore_error(exc_info[1]): | ||
| with sentry_sdk.push_scope() as scope: | ||
| scope.set_tag("action", "ignored") |
There was a problem hiding this comment.
Love this use of tagging
|
|
||
| assert job.is_failed | ||
|
|
||
| # TODO: Determine why the `store_task_exception_handler` is not called. |
There was a problem hiding this comment.
Is this a blocker or OK as a known-unknown?
There was a problem hiding this comment.
I added more tests so there's 100% coverage, it's just broken into separate tests. But I'd like to solve this riddle at some point. So I don't consider it a blocker, just a hard thing to test.
robhudson
force-pushed
the
rq-queues
branch
5 times, most recently
from
5bd0ea3 to
d748b36
Compare
|
|
||
|
|
||
| def rq_on_success(job, connection, result, *args, **kwargs): | ||
| # Don't fire statsd metrics in maintenance mode. |
There was a problem hiding this comment.
In maintenance mode we're going to queue the task in the db rather than do the task, so we'd get timings for doing the db insert rather than timings for the task, which would confuse the timings. And a "success" for the db insert isn't the same as a "success" for the task.
robhudson
force-pushed
the
rq-queues
branch
6 times, most recently
from
344f160 to
f7276ad
Compare
robhudson
force-pushed
the
rq-queues
branch
2 times, most recently
from
5afe1e1 to
bf373e5
Compare
In mozmeao/basket#1069 Basket tasks were transitionned to rq This pins Basket to its previous version until we figure out how to migrate our integration test setup.
In mozmeao/basket#1069 Basket tasks were transitionned to rq This pins Basket to its previous version until we figure out how to migrate our integration test setup.
Still some work to be done here...
[CronJob](https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/)REDIS_URLalso Django settings, rather than env variables.