fix: dagster concurrency priority

[Jabolol]

No description provided.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
kariba-network	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Sep 19, 2024 6:16pm
oso-www	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Sep 19, 2024 6:16pm

[Jabolol]

After limiting concurrent run slots to 1 in commit 6b28178, I’ve come up with the following solution. What do you think, @ravenac95?

1. Stop using the command dagster instance concurrency set $tag $value.

2. Set a default concurrency limit for all ops to 1 by adding this configuration:

concurrency:
  default_op_concurrency_limit: 1

Warning

This config is mentioned in the documentation but is not present in Dagster's helm YAML schema. We should confirm whether this is the correct way to set the default concurrency limit.

3. Add the tag dagster/concurrency_key with value $op_name to all partitioned assets/jobs (e.g., open_collective_expenses, eas_optimism) to ensure that the concurrency limit is applied at the op level, allowing only 1 concurrent run per op by default.

This approach allows different ops to run concurrently while enforcing a limit of one concurrent run for each individual op. With the buffer set to 1 (from my last commit), we also avoid spinning up unnecessary pods that would take up space and block other runs.

[ravenac95]

After limiting concurrent run slots to 1 in commit 6b28178, I’ve come up with the following solution. What do you think, @ravenac95?

1. Stop using the command dagster instance concurrency set $tag $value.
2. Set a default concurrency limit for all ops to 1 by adding this configuration:

concurrency:
  default_op_concurrency_limit: 1

Warning

This config is mentioned in the documentation but is not present in Dagster's helm YAML schema. We should confirm whether this is the correct way to set the default concurrency limit.

3. Add the tag dagster/concurrency_key with value $op_name to all partitioned assets/jobs (e.g., open_collective_expenses, eas_optimism) to ensure that the concurrency limit is applied at the op level, allowing only 1 concurrent run per op by default.

This approach allows different ops to run concurrently while enforcing a limit of one concurrent run for each individual op. With the buffer set to 1 (from my last commit), we also avoid spinning up unnecessary pods that would take up space and block other runs.

Ah! This is great. I didn't realize it limited ops of the same name haha. Let's do it.

[Jabolol]

Ah! This is great. I didn't realize it limited ops of the same name haha. Let's do it.

Unfortunately, this property isn't supported in the current Helm setup (see dagster-io/dagster#19597, inactive since February 2024). As a workaround, we'll use tag_concurrency_limits with the existing dagster/concurrency_key tag, enabling applyLimitPerUniqueValue and setting limit to 1 to achieve the same result, as this configuration is supported.

We are able to limit the runs to 1 per backfill, and queue the rest, without interfering with all other materialisations.

[Jabolol]

/test-deploy 3c17301

[oso-prs]

Test deployment unnecessary, no dbt files have been changed.