Parallel job creation

[DrJosh9000]

What

For each poll of BK for jobs, run the rest of the pipeline (deduper -> limiter -> scheduler) in parallel (with a fairly conservative default concurrency of 5 goroutines).
Randomise the order in which jobs are sent.

Why

If 100% of jobs sent to k8s successfully create, then (up to the MaxInFlight limit if it applies) the current process should always make progress even if the k8s API was slow, because staleCtx only interrupts waiting for limiter tokens, not the creation of jobs. (Any job that makes its way through the limiter will be sent to k8s by the scheduler.) The other bits in the middle shouldn't be so slow that staleCtx is cancelled first.

If some jobs successfully create and some won't, then that's most likely a property of the job (e.g. the job name collides with an existing job). If we try and fail to create the same job over and over, and creating jobs is slow enough that we don't get through enough of the broken ones before staleCtx expires, we won't progress.

Randomising the order of creating jobs means jobs that could successfully be created get a chance. Adding parallelism to create more jobs at a time will increase load on the k8s cluster, but let more jobs be tried before staleness kicks in.

[wolfeidau]

Nifty error switch 🤔 👍🏻 🚀