Add logic to recover from quarantined and completed states #13071
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
area/platform
lmossman
commented
May 21, 2022
Comment on lines
+184
to
+186
| // A non-deleted workflow being in a COMPLETED state is unexpected, and should be corrected | ||
| throw new UnreachableWorkflowException( | ||
| String.format("ConnectionManagerWorkflow for connection %s is unreachable due to having COMPLETED status.", connectionId)); |
There was a problem hiding this comment.
Added this case because when testing I saw an occurrence of a workflow with COMPLETED status that was not deleted. Since it was COMPLETED, it's state could still be queried, but no signals could be executed on it. This logic corrects this by correctly treating workflows in this state as "unreachable"
benmoriceau
approved these changes
May 23, 2022
| } catch (final Exception e) { | ||
| log.warn( | ||
| "Could not terminate temporal workflow due to the following error; " | ||
| + "this may be because there is currently running workflow for this connection.", |
There was a problem hiding this comment.
Do you mean "no running workflow" here
There was a problem hiding this comment.
Yes I did, good catch
jscottpolevault
pushed a commit
to jscottpolevault/airbyte
that referenced
this pull request
Jun 1, 2022
…#13071) * add logic to recover from quarantined and completed states * move status retrieval into try-catch * fix typo in log * add one more tests * mvoe isWorkflowStateRunning into ConnectionManagerUtils to be more direct * format
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
Resolves #10932
How
Adds logic to handle quarantined and "completed" states of workflows, by detecting workflows in this state and properly reporting them as "unreachable". This PR also adds logic to attempt termination of a workflow before restarting it. This is important for cases like the quarantined case where we cannot start a fresh workflow for the connection until we have terminated the existing one.
Recommended reading order
x.javay.python