Proposal: Support concurrency groups

[irvinlim]

Motivation

Currently, concurrency policies are only supported only on a single JobConfig-level. There could be two JobConfigs that cannot run at the same time, and we want to provide some concurrency control on a controller-level to help with this.

Proposal

It should be simple to extend concurrency policy to support named concurrency groups, such that:

1. The default concurrency group is empty, which means to group by JobConfig (predominant behavior).
2. If so desired, users can specify an optional concurrency.group. This group name is namespaced, and two identical names in different namespaces should not contribute to the same concurrency count.
3. Need to enforce concurrency policies for this group in multiple places:
   • ActiveJobStore: Need to use group instead of JobConfig name.
   • JobQueueController: Probably need to rewrite to update PerConfigReconciler to PerGroupReconciler.
   • JobValidatingWebhook: Need to index all Jobs by group, instead of looking up JobConfig's status for enforcement.
   • JobConfigController: Need to handle new JobConfigStatus.
4. Concurrency group name can support context variables for the jobconfig context.
   • Example: The default concurrency group would be ${jobconfig.uid}, which evaluates to the JobConfig's UID.

API Design

JobConfig

1. Update ConcurrencySpec like as follows:

   apiVersion: execution.furiko.io/v1alpha1
   kind: JobConfig
   metadata:
     name: my-job-config
     namespace: my-namespace
   spec:
     concurrency:
       policy: Forbid
       group: my-custom-group

2. JobConfigStatus will have to be updated as well:

   apiVersion: execution.furiko.io/v1alpha1
   kind: JobConfig
   metadata:
     name: my-job-config
     namespace: my-namespace
   status:
     state: JobQueued  # Refers to jobs for the JobConfig, not the concurrency group.
     jobs:  # Contains references for JobConfig's actual children jobs.
       queued: 2
       queuedJobs: 
         - namespace: my-namespace
            name: my-job-config.1
         - namespace: my-namespace
            name: my-job-config.2
     concurrencyGroup:  # Contains references for all Jobs in the JobConfig's concurrency group.
       name: my-custom-group  # Defaults to JobConfig name
       queued: 4
       queuedJobs: 
         - namespace: my-namespace
            name: my-job-config.1
         - namespace: my-namespace
            name: my-job-config.2
         - namespace: my-namespace
            name: another-job-config.1
         - namespace: my-namespace
            name: another-job-config.1

Job

1. Update StartPolicySpec as follows:

   apiVersion: execution.furiko.io/v1alpha1
   kind: Job
   metadata:
     name: my-job
     namespace: my-namespace
   spec:
     startPolicy:
       concurrencyPolicy: Forbid
       concurrencyGroup: my-custom-group

2. It should be possible for the JobStatus to show a more detailed message about the concurrency group:

   apiVersion: execution.furiko.io/v1alpha1
   kind: Job
   metadata:
     name: my-job
     namespace: my-namespace
   status:
     condition:
       finished:
         result: AdmissionError
         reason: AdmissionError
         message: Cannot start new Job, concurrency group "my-custom-group" has 2 active Jobs but concurrency policy is Forbid

References

• GitHub Actions: https://docs.github.com/en/actions/using-jobs/using-concurrency
• Argo Workflows: https://argoproj.github.io/argo-workflows/synchronization/