Break sequential loop if an iteration fails

[laymain]

Summary

Break sequential loop if an iteration fails

Motivation

I have a sequential loop using parallelism: 1 and I need to stop the loop execution when an iteration fails,
in order to not process next iteration if previous one has failed.

Given this template:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: seq-loop-
spec:
  arguments:
    parameters:
    - name: items
      value: |
        ["a", "b", "c"]

  entrypoint: seq-loop
  templates:
  - name: seq-loop
    parallelism: 1
    inputs:
      parameters:
      - name: items
    steps:
    - - name: iteration
        template: iteration
        withParam: "{{inputs.parameters.items}}"

  - name: iteration
    steps:
    - - name: step1
        template: succeed-step
    - - name: step2
        template: failed-step

  - name: succeed-step
    container:
      image: alpine
      command: ['/bin/sh', '-c']
      args: ["exit 0"]

  - name: failed-step
    container:
      image: alpine
      command: ['/bin/sh', '-c']
      args: ["exit 1"]

Currently, executing this workflow gives:

STEP                   TEMPLATE      PODNAME                    DURATION  MESSAGE
 ✖ seq-loop-b68gn      seq-loop                                           child 'seq-loop-b68gn-342633600' failed
 └-·-✖ iteration(0:a)  iteration                                          child 'seq-loop-b68gn-3536970033' failed
   | ├---✔ step1       succeed-step  seq-loop-b68gn-2592104941  9s
   | └---✖ step2       failed-step   seq-loop-b68gn-3536970033  4s        failed with exit code 1
   ├-✖ iteration(1:b)  iteration                                          child 'seq-loop-b68gn-1741679919' failed
   | ├---✔ step1       succeed-step  seq-loop-b68gn-3000376863  4s
   | └---✖ step2       failed-step   seq-loop-b68gn-1741679919  4s        failed with exit code 1
   └-✖ iteration(2:c)  iteration                                          child 'seq-loop-b68gn-221656217' failed
     ├---✔ step1       succeed-step  seq-loop-b68gn-1906215637  4s
     └---✖ step2       failed-step   seq-loop-b68gn-221656217   4s        failed with exit code 1

I would like to be able to break the loop if an iteration fails:

STEP                   TEMPLATE      PODNAME                    DURATION  MESSAGE
 ✖ seq-loop-b68gn      seq-loop                                           child 'seq-loop-b68gn-342633600' failed
 └-·-✖ iteration(0:a)  iteration                                          child 'seq-loop-b68gn-3536970033' failed
     ├---✔ step1       succeed-step  seq-loop-b68gn-2592104941  9s
     └---✖ step2       failed-step   seq-loop-b68gn-3536970033  4s        failed with exit code 1

---

Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

[simster7]

When you use withParam Argo creates and schedules all resulting tasks in parallel. Since you have parallelism: 1, they do in practice run sequentially, but at that point they have already been scheduled to run, regardless of failures.

Perhaps using this recursive for loop example might work better for this use case?

[jessesuen]

I believe we used to fail fast when parallelism was reduced. This might be a regression.

[laymain]

@simster7 I could indeed use recursive for loop, but I have hundreds of iteration which I feel will end
with an unnecessary over-complicated workflow result.
I hoped that there was another option than using recursive for loop.

[simster7]

I'll investigate @jessesuen's comment that this is in fact a regression and it should actually be possible to have the behavior you desired using withParam and parallelism

[laymain]

Thank you for digging into that issue

[simster7]

I took a look at the code and I don't get any indication that this behavior should actually be possible. I will discuss with @jessesuen and report back

[simster7]

I discussed this with Jesse and tried this in 2.5 (https://github.com/argoproj/argo/blob/release-2.5/workflow/controller/steps.go) and we determined that this was never possible, so it is in fact an enhancement.

We've determined that this is something we would consider, we would just need to come up with a design that would allow the user to specify that they would like this behavior when using with* and parallelism.

[laymain]

I would really appreciate this feature.
I am actually using the recursive loop workaround, but it gives a way more complicated workflow.

[simster7]

@laymain Please add any feedback you may have at #5315

[simster7]

This is now supported by setting failFast: true in the template that invokes the items/has a parallelism: 1

  - name: seq-loop
    failFast: true
    inputs:
      parameters:
      - name: items
    parallelism: 1
    steps:
    - - name: iteration
        template: iteration
        withParam: '{{inputs.parameters.items}}'

This will be out on v3.1

[simster7]

Tagging upvoters: @tanenbaum @ppapasani1 @jbehling @oliverwm1 @shbalaku @trim777 @bybybybyb @dinever @rajasudhan @raph-af @camilocot

[laymain]

Thank you for the update, that seems nice!

Can you confirm that it will work with a parallelism that have any value?
Let's say that we have a parallelism: 3 and failFast: true, if one iteration fails every next iteration (that aren't running yet) will fail?

Also, is there a way to test the v3.1 at its current stage?

[dinever]

Thank you for the update, that seems nice!

Can you confirm that it will work with a parallelism that have any value?
Let's say that we have a parallelism: 3 and failFast: true, if one iteration fails every next iteration (that aren't running yet) will fail?

Also, is there a way to test the v3.1 at its current stage?

Yes. If we have parallelism: 3 and failFast: true, if one iteration fails, the entire sequential loop will fail fast, and the iterations that are not running yet will not be triggered.

Try this. We have 5 iterations and parallelism: 3. In this case, the first 3 iterations will run together. The next 2 iterations will not run.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: seq-loop-
spec:
  arguments:
    parameters:
    - name: items
      value: |
        ["a", "b", "c", "d", "e"]

  entrypoint: seq-loop
  templates:
  - name: seq-loop
    failFast: true
    parallelism: 3
    inputs:
      parameters:
      - name: items
    steps:
    - - name: iteration
        template: iteration
        withParam: "{{inputs.parameters.items}}"

  - name: iteration
    steps:
    - - name: step1
        template: succeed-step
    - - name: step2
        template: failed-step

  - name: succeed-step
    container:
      image: alpine
      command: ['/bin/sh', '-c']
      args: ["exit 0"]

  - name: failed-step
    container:
      image: alpine
      command: ['/bin/sh', '-c']
      args: ["exit 1"]