Inform the User when temporal quarantines a job

[supertopher]

Wut

Temporal has a quarantine state. This state is to reduce overloading the system with a large pile of failed requests. It's a good idea, we want this to happen.

Temporal will attempt to run a job 10 times. If temporal does not receive an ACK for the start of the job for all 10 retries that job goes into quarantine.

From the user's perspective this can look like a single sync attempt was fired and that attempt in still in progress.

To unstick the job, an engineer must run tctl

How to do this

¯_(ツ)_/¯

TLDR you figure it out

We could have a time based check from airbyte to check with temporal on current job status and inform the user for non-running state

We could check state on every page reload and avoid the whole timing thing

Some way smarter way you just though of while reading my silly suggestions

You will know you are done when

• Users are aware their jobs are not running in the UI when temporal knows they are not running due to quarantine state
• Our database reflects that jobs temporal knows are in a quarantine state are in fact quarantined

[supertopher]

@benmoriceau @lmossman please edit this or comment if my understanding of the world here is incorrect or you see a better way

[davinchia]

We no longer quarantine jobs as of #13779.