Dagster concurrency limits preventing some jobs from starting

[ravenac95]

What is it?

Right now the max_concurrency limit for the queued run coordinator on dagster is set to 30, but we did this in hopes to limit concurrency for things like the open_collective backfill. Upon fixing issues with the k8s_job_executor and adding the dagster/concurrency_key on a per op basis we've been able to properly get things running with concurrency limits. However, because the max_concurrency value is set, dagster actually ends up spinning up pods for each pending run up to the max_concurrency value. The reason it does this is because these "runs" in dagster are used to monitor and trigger execution of a run's "steps." The steps are the actual asset/op code executing while the run is just a coordinator process. This is problematic because these pods are taking up unnecessary resources that are basically "sleeping" because their steps are being blocked by the dagster/concurrency_key. This then prevents new jobs from starting that are not blocked by the same concurrency_key.

Here are some potential paths forward:

• Remove the max_concurrency limit. This has issues because that means if we queued 500 runs then 500 pods would be started to manage those runs.
• To overcome that issue the dagster docs seems to point to this: https://docs.dagster.io/guides/limiting-concurrency-in-data-pipelines#throttling-concurrency-limited-runs
• We may also want to look at introducing tag_concurrency_limits for jobs. Though this will encounter the issues we originally found when attempting to limit concurrency in that everything must be a job.
• Set dagster/priority for our important daily jobs to be -1. This is a quick fix for now and should likely be done immediately.

[ravenac95]

@Jabolol this is some detail on the problem I saw. I assigned both of us mostly to whomever gets parts/all of this done first. I imagine one of us may need it sooner (likely me :D)