fix(controller): Use node.Name instead of node.DisplayName for onExit nodes #5486

simster7 · 2021-03-22T19:07:32Z

Split from #5478

Signed-off-by: Simon Behar <simbeh7@gmail.com>

simster7 · 2021-03-22T19:12:31Z

workflow/controller/steps.go

@@ -260,7 +260,7 @@ func (woc *wfOperationCtx) executeStepGroup(ctx context.Context, stepGroup []wfv
 		if !childNode.Fulfilled() {
 			completed = false
 		} else if childNode.Completed() {
-			hasOnExitNode, onExitNode, err := woc.runOnExitNode(ctx, step.OnExit, step.Name, childNode.Name, stepsCtx.boundaryID, stepsCtx.tmplCtx)
+			hasOnExitNode, onExitNode, err := woc.runOnExitNode(ctx, step.OnExit, childNode.Name, stepsCtx.boundaryID, stepsCtx.tmplCtx)


parentDisplayName is non unique, causing nodes with the same display name (e.g. when using withParam) to create onExit nodes of the same name. After the first onExit node is created, subsequent onExit nodes created by different parent nodes (but with the same display name) will fail creation as they already exist. Use parentNodeName which is always unique instead.

codecov · 2021-03-22T19:14:03Z

Codecov Report

Merging #5486 (885c05c) into master (3065941) will increase coverage by 0.20%.
The diff coverage is 94.73%.

❗ Current head 885c05c differs from pull request most recent head 63dec32. Consider uploading reports for the commit 63dec32 to get more accurate results

@@            Coverage Diff             @@
##           master    #5486      +/-   ##
==========================================
+ Coverage   46.54%   46.74%   +0.20%     
==========================================
  Files         240      240              
  Lines       15001    15004       +3     
==========================================
+ Hits         6982     7014      +32     
+ Misses       7119     7091      -28     
+ Partials      900      899       -1

Impacted Files	Coverage Δ
workflow/common/util.go	`8.10% <0.00%> (ø)`
pkg/apis/workflow/v1alpha1/workflow_types.go	`45.78% <100.00%> (+0.25%)`	⬆️
util/errors/errors.go	`100.00% <100.00%> (ø)`
workflow/controller/dag.go	`72.67% <100.00%> (+0.31%)`	⬆️
workflow/controller/operator.go	`70.69% <100.00%> (+0.43%)`	⬆️
pkg/apiclient/apiclient.go	`5.55% <0.00%> (-4.45%)`	⬇️
cmd/argo/commands/get.go	`56.66% <0.00%> (ø)`
pkg/apiclient/http1-client.go	`0.00% <0.00%> (ø)`
pkg/apiclient/logs-intermediary.go	`0.00% <0.00%> (ø)`
pkg/apiclient/http1/event-watch-client.go	`0.00% <0.00%> (ø)`
... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3065941...63dec32. Read the comment docs.

alexec · 2021-03-22T20:21:05Z

what happens to in-flight workflows at the time of upgrade? can do they end up with 2x on-exit nodes?

simster7 · 2021-03-22T20:52:01Z

what happens to in-flight workflows at the time of upgrade? can do they end up with 2x on-exit nodes?

Yes, there is a window of time where onExit nodes could get duplicated in the event of an upgrade while a cluster is running. note this window is only from the time an onExit node with the old name is created until the time its boundary node (Step Group, DAG) completes. Once the boundary node completes, we won't recurse down to the onExit node again.

I don't believe we make guarantees on the consistency of a running workflow if there is a version upgrade while it's running.

alexec · 2021-03-22T21:05:48Z

I don't think many users expect an upgrade to typically break workflows. Is there an alternative you can see?

simster7 · 2021-03-22T21:26:25Z

I don't think many users expect an upgrade to typically break workflows. Is there an alternative you can see?

@alexec I don't believe so. This is a narrow issue: are we using the right name or not? We are currently not using the right name, which means there are bugs in the code (see #5486 (comment)).

If it's critical that no running workflows are broken during the upgrade, I could introduce some code that:

first checks if a node exists with the old name
if it does, proceed with the old name
if it doesn't, proceed with the new name

We'd have to keep this scaffold code in the codebase for a couple of versions to ensure that users upgrading and skipping some patch versions (e.g. v3.0.1 -> v3.0.3) are also not affected. We'd then remove the scaffold after a "tolerance period". (How would we determine the length of that tolerance period?)

simster7 · 2021-03-22T21:32:49Z

Also perhaps we could include a notice in the release notes the advise users to not upgrade with running workflows. And we can include that notice whenever applicable. It seems to me that we must have already made similar bug fixes that would have broken running nodes in the past.

This could be a good middle ground between users not seeing unexpected, one-time issues during upgraded and us being able to fix bugs that currently exist in the codebase.

simster7 · 2021-03-23T18:30:30Z

Decision: add scaffolding code

Signed-off-by: Simon Behar <simbeh7@gmail.com>

simster7

@alexec Now fully backwards compatible, with tests that ensure it. Ready for another review 🙂

simster7 · 2021-03-24T15:43:49Z

workflow/controller/controller_test.go

@@ -301,6 +301,31 @@ func withAnnotation(key, val string) with {
 	return func(pod *apiv1.Pod) { pod.Annotations[key] = val }
 }

+// createRunningPods creates the pods that are marked as running in a given test so that they can be accessed by the
+// pod assessor
+func createRunningPods(ctx context.Context, woc *wfOperationCtx, with ...with) {


Useful utility testing function to create pods in the client that are supposed to be running according to the test workflow status.

unused? delete?

Only the with parameter was unused. Removed.

simster7 · 2021-03-24T15:44:18Z

workflow/controller/dag.go

+		// Previously we used `depNode.DisplayName` to generate all onExit node names. However, as these can be non-unique
+		// we transitioned to using `depNode.Name` instead, which are guaranteed to be unique. In order to not disrupt
+		// running workflows during upgrade time, we do an additional check to see if there is an onExit node with the old
+		// name (`depNode.DisplayName`).
+		// TODO: This scaffold code should be removed after a couple of "grace period" version upgrades to allow transitions. It was introduced in v3.0.0
+		// See more: https://github.com/argoproj/argo-workflows/issues/5502
+		legacyOnExitNodeName := common.GenerateOnExitNodeName(depNode.DisplayName)
+		if onExitNode := d.wf.GetNodeByName(legacyOnExitNodeName); onExitNode != nil && d.wf.GetNodeByName(depNode.Name).HasChild(onExitNode.ID) {


This is tested by TestOnExitDAGStatusCompatibility in this PR

simster7 · 2021-03-24T15:44:59Z

workflow/controller/operator_test.go

@@ -6314,3 +6314,223 @@ func TestGenerateOutputResultRegex(t *testing.T) {
 	assert.Equal(t, `steps\.template-name\.outputs\.result`, ref)
 	assert.Equal(t, `steps\[['\"]template-name['\"]\]\.outputs.result`, expr)
 }
+
+const testOnExitNameBackwardsCompatibility = `
+apiVersion: argoproj.io/v1alpha1


I've already ensured these test workflows are minimal.

Signed-off-by: Simon Behar <simbeh7@gmail.com>

simster7 · 2021-03-24T15:57:50Z

workflow/controller/operator.go

+		// Previously we used `parentDisplayName` to generate all onExit node names. However, as these can be non-unique
+		// we transitioned to using `parentNodeName` instead, which are guaranteed to be unique. In order to not disrupt
+		// running workflows during upgrade time, we first check if there is an onExit node that currently exists with the
+		// legacy name AND said node is a child of the parent node. If it does, we continue execution with the legacy name.
+		// If it doesn't, we use the new (and unique) name for all operations henceforth.
+		// TODO: This scaffold code should be removed after a couple of "grace period" version upgrades to allow transitions. It was introduced in v3.0.0
+		// When the scaffold code is removed, we should only have the following:
+		//
+		// 		onExitNodeName := common.GenerateOnExitNodeName(parentNodeName)
+		//
+		// See more: https://github.com/argoproj/argo-workflows/issues/5502
+		onExitNodeName := common.GenerateOnExitNodeName(parentNodeName)
+		legacyOnExitNodeName := common.GenerateOnExitNodeName(parentDisplayName)
+		if legacyNameNode := woc.wf.GetNodeByName(legacyOnExitNodeName); legacyNameNode != nil && woc.wf.GetNodeByName(parentNodeName).HasChild(legacyNameNode.ID) {
+			onExitNodeName = legacyOnExitNodeName
+		}


This is tested by TestOnExitNameBackwardsCompatibility in this PR

the block of code looks to be duplicated - maybe create a utility func?

I had a utility function before, but once all the parameters were taken into account it didn't really reduce complexity so I decided to keep it like so.

alexec · 2021-03-24T16:48:37Z

pkg/apis/workflow/v1alpha1/workflow_types.go

@@ -1806,6 +1806,15 @@ func (n NodeStatus) GetDuration() time.Duration {
 	return n.FinishedAt.Sub(n.StartedAt.Time)
 }

+func (n NodeStatus) HasChild(childID string) bool {
+	for _, nodeID := range n.Children {


unit test please

Working on it right now, will push soon

alexec · 2021-03-24T16:49:23Z

workflow/controller/controller_test.go

@@ -301,6 +301,31 @@ func withAnnotation(key, val string) with {
 	return func(pod *apiv1.Pod) { pod.Annotations[key] = val }
 }

+// createRunningPods creates the pods that are marked as running in a given test so that they can be accessed by the
+// pod assessor
+func createRunningPods(ctx context.Context, woc *wfOperationCtx, with ...with) {


unused? delete?

alexec · 2021-03-24T16:49:56Z

workflow/controller/dag.go

+		// Previously we used `depNode.DisplayName` to generate all onExit node names. However, as these can be non-unique
+		// we transitioned to using `depNode.Name` instead, which are guaranteed to be unique. In order to not disrupt
+		// running workflows during upgrade time, we do an additional check to see if there is an onExit node with the old
+		// name (`depNode.DisplayName`).
+		// TODO: This scaffold code should be removed after a couple of "grace period" version upgrades to allow transitions. It was introduced in v3.0.0
+		// See more: https://github.com/argoproj/argo-workflows/issues/5502
+		legacyOnExitNodeName := common.GenerateOnExitNodeName(depNode.DisplayName)
+		if onExitNode := d.wf.GetNodeByName(legacyOnExitNodeName); onExitNode != nil && d.wf.GetNodeByName(depNode.Name).HasChild(onExitNode.ID) {


alexec · 2021-03-24T16:50:26Z

workflow/controller/operator.go

+		// Previously we used `parentDisplayName` to generate all onExit node names. However, as these can be non-unique
+		// we transitioned to using `parentNodeName` instead, which are guaranteed to be unique. In order to not disrupt
+		// running workflows during upgrade time, we first check if there is an onExit node that currently exists with the
+		// legacy name AND said node is a child of the parent node. If it does, we continue execution with the legacy name.
+		// If it doesn't, we use the new (and unique) name for all operations henceforth.
+		// TODO: This scaffold code should be removed after a couple of "grace period" version upgrades to allow transitions. It was introduced in v3.0.0
+		// When the scaffold code is removed, we should only have the following:
+		//
+		// 		onExitNodeName := common.GenerateOnExitNodeName(parentNodeName)
+		//
+		// See more: https://github.com/argoproj/argo-workflows/issues/5502
+		onExitNodeName := common.GenerateOnExitNodeName(parentNodeName)
+		legacyOnExitNodeName := common.GenerateOnExitNodeName(parentDisplayName)
+		if legacyNameNode := woc.wf.GetNodeByName(legacyOnExitNodeName); legacyNameNode != nil && woc.wf.GetNodeByName(parentNodeName).HasChild(legacyNameNode.ID) {
+			onExitNodeName = legacyOnExitNodeName
+		}


the block of code looks to be duplicated - maybe create a utility func?

Signed-off-by: Simon Behar <simbeh7@gmail.com>

… nodes (#5486) Signed-off-by: Simon Behar <simbeh7@gmail.com>

fix: Use node.Name instead of node.DisplayName for onExit nodes

1fdc86f

Signed-off-by: Simon Behar <simbeh7@gmail.com>

simster7 mentioned this pull request Mar 22, 2021

fix: Create new boundary to run onExit nodes #5478

Closed

simster7 commented Mar 22, 2021

View reviewed changes

terrytangyuan approved these changes Mar 22, 2021

View reviewed changes

simster7 requested a review from alexec March 22, 2021 20:06

simster7 added 3 commits March 24, 2021 07:47

scaffold

db33dc3

Signed-off-by: Simon Behar <simbeh7@gmail.com>

prog

3eb964c

Signed-off-by: Simon Behar <simbeh7@gmail.com>

test

fe21678

Signed-off-by: Simon Behar <simbeh7@gmail.com>

simster7 commented Mar 24, 2021

View reviewed changes

Merge branch 'master' into oe-name

882ebea

simster7 mentioned this pull request Mar 24, 2021

Remove onExit scaffolding code #5502

Closed

simster7 added 2 commits March 24, 2021 08:56

done

3db7f9b

Signed-off-by: Simon Behar <simbeh7@gmail.com>

done

bc9ad98

Signed-off-by: Simon Behar <simbeh7@gmail.com>

simster7 commented Mar 24, 2021

View reviewed changes

simster7 assigned alexec Mar 24, 2021

alexec reviewed Mar 24, 2021

View reviewed changes

alexec requested changes Mar 24, 2021

View reviewed changes

alexec changed the title ~~fix: Use node.Name instead of node.DisplayName for onExit nodes~~ fix(controller): Use node.Name instead of node.DisplayName for onExit nodes Mar 24, 2021

alexec approved these changes Mar 24, 2021

View reviewed changes

tests

63dec32

Signed-off-by: Simon Behar <simbeh7@gmail.com>

simster7 merged commit d964fe4 into argoproj:master Mar 24, 2021

This was referenced Mar 29, 2021

v3.0.0-rc10 cherry-pick #5541

Closed

v2.12.11 cherry-pick #5542

Closed

simster7 added a commit that referenced this pull request Mar 30, 2021

fix(controller): Use node.Name instead of node.DisplayName for onExit…

c77f1ec

… nodes (#5486) Signed-off-by: Simon Behar <simbeh7@gmail.com>

simster7 mentioned this pull request Apr 19, 2021

v3.0.2 cherry-pick #5715

Closed

50 tasks

simster7 mentioned this pull request Jul 7, 2021

chore: Remove onExit naming transition scaffolding code #6297

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(controller): Use node.Name instead of node.DisplayName for onExit nodes #5486

fix(controller): Use node.Name instead of node.DisplayName for onExit nodes #5486

simster7 commented Mar 22, 2021 •

edited

Loading

simster7 Mar 22, 2021

codecov bot commented Mar 22, 2021 •

edited

Loading

alexec commented Mar 22, 2021 •

edited

Loading

simster7 commented Mar 22, 2021

alexec commented Mar 22, 2021

simster7 commented Mar 22, 2021

simster7 commented Mar 22, 2021

simster7 commented Mar 23, 2021

simster7 left a comment

simster7 Mar 24, 2021

alexec Mar 24, 2021

simster7 Mar 24, 2021

simster7 Mar 24, 2021

alexec Mar 24, 2021

simster7 Mar 24, 2021

simster7 Mar 24, 2021

alexec Mar 24, 2021

simster7 Mar 24, 2021

alexec Mar 24, 2021

simster7 Mar 24, 2021

alexec Mar 24, 2021

alexec Mar 24, 2021

alexec Mar 24, 2021

fix(controller): Use node.Name instead of node.DisplayName for onExit nodes #5486

fix(controller): Use node.Name instead of node.DisplayName for onExit nodes #5486

Conversation

simster7 commented Mar 22, 2021 • edited Loading

Choose a reason for hiding this comment

codecov bot commented Mar 22, 2021 • edited Loading

Codecov Report

alexec commented Mar 22, 2021 • edited Loading

simster7 commented Mar 22, 2021

alexec commented Mar 22, 2021

simster7 commented Mar 22, 2021

simster7 commented Mar 22, 2021

simster7 commented Mar 23, 2021

simster7 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

simster7 commented Mar 22, 2021 •

edited

Loading

codecov bot commented Mar 22, 2021 •

edited

Loading

alexec commented Mar 22, 2021 •

edited

Loading