workflow - argoexec container stuck on wait after main container finishes #7480

m0nk3y-s3c · 2022-01-01T12:42:41Z

m0nk3y-s3c
Jan 1, 2022

Hi everyone, does anyone came across this issue?
#6064
I have the same thing, tried now to run https://raw.githubusercontent.com/argoproj/argo-workflows/master/examples/steps.yaml and the quay.io/argoproj/argoexec:v3.2.6 image is in running mode even when my main docker/whalesay finished, the workflow never ends...
using GKE
this is my argo server version https://github.com/argoproj/argo-workflows/releases/download/v3.2.6/install.yaml
argo events install https://raw.githubusercontent.com/argoproj/argo-events/master/manifests/install.yaml
argo event bus https://raw.githubusercontent.com/argoproj/argo-events/master/examples/eventbus/native.yaml
webhook event source https://raw.githubusercontent.com/argoproj/argo-events/stable/examples/event-sources/webhook.yaml
sensor https://raw.githubusercontent.com/argoproj/argo-events/stable/examples/sensors/webhook.yaml

Answered by terrytangyuan

Jan 3, 2022

Please provide additional information as mentioned in the issue template: https://github.com/argoproj/argo-workflows/blob/master/.github/ISSUE_TEMPLATE/bug_report.md

View full answer

terrytangyuan · 2022-01-03T15:41:38Z

terrytangyuan
Jan 3, 2022
Maintainer

Please provide additional information as mentioned in the issue template: https://github.com/argoproj/argo-workflows/blob/master/.github/ISSUE_TEMPLATE/bug_report.md

3 replies

m0nk3y-s3c Jan 7, 2022
Author

Please provide additional information as mentioned in the issue template: https://github.com/argoproj/argo-workflows/blob/master/.github/ISSUE_TEMPLATE/bug_report.md

Hi @terrytangyuan hope all is well :)

Should I report an issue or this is good enough for now?

terrytangyuan Jan 7, 2022
Maintainer

Thanks. Looks like this is more suitable in a GitHub issue that's easier for us to track and others to chime in. Do you mind moving this to an issue?

m0nk3y-s3c Jan 8, 2022
Author

@terrytangyuan Got it.

#7530

m0nk3y-s3c · 2022-01-05T17:14:31Z

m0nk3y-s3c
Jan 5, 2022
Author

Thanks @terrytangyuan, here it is

Summary

What happened/what you expected to happen?
Installed the stpes.yaml from example, but after the first step in finished the wait container get stuck and never get to step2.

What version of Argo Workflows are you running?
v3.2.6

Diagnostics

Either a workflow that reproduces the bug, or paste you whole workflow YAML, including status, something like:

kubectl get wf -o yaml ${workflow}
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  annotations:
    workflows.argoproj.io/pod-name-format: v1
  creationTimestamp: "2022-01-05T17:02:35Z"
  generateName: steps-
  generation: 4
  labels:
    workflows.argoproj.io/phase: Running
  name: steps-qcjw9
  namespace: argo
  resourceVersion: "3880"
  uid: 055699a2-c494-41df-9eda-62ed206661e6
spec:
  arguments:
    parameters:
    - name: message
      value: HelpMeSolveThis
  entrypoint: hello-hello-hello
  templates:
  - inputs: {}
    metadata: {}
    name: hello-hello-hello
    outputs: {}
    steps:
    - - arguments:
          parameters:
          - name: message
            value: hello1
        name: hello1
        template: whalesay
    - - arguments:
          parameters:
          - name: message
            value: hello2a
        name: hello2a
        template: whalesay
      - arguments:
          parameters:
          - name: message
            value: hello2b
        name: hello2b
        template: whalesay
  - container:
      args:
      - '{{inputs.parameters.message}}'
      command:
      - cowsay
      image: docker/whalesay
      name: ""
      resources: {}
    inputs:
      parameters:
      - name: message
    metadata: {}
    name: whalesay
    outputs: {}
status:
  artifactRepositoryRef:
    artifactRepository: {}
    default: true
  conditions:
  - status: "True"
    type: PodRunning
  finishedAt: null
  nodes:
    steps-qcjw9:
      children:
      - steps-qcjw9-3301525239
      displayName: steps-qcjw9
      finishedAt: null
      id: steps-qcjw9
      name: steps-qcjw9
      phase: Running
      progress: 0/1
      startedAt: "2022-01-05T17:02:35Z"
      templateName: hello-hello-hello
      templateScope: local/steps-qcjw9
      type: Steps
    steps-qcjw9-2094437448:
      boundaryID: steps-qcjw9
      displayName: hello1
      finishedAt: null
      hostNodeName: gke-bug-bounty-argo-default-pool-6465c554-06m1
      id: steps-qcjw9-2094437448
      inputs:
        parameters:
        - name: message
          value: hello1
      name: steps-qcjw9[0].hello1
      phase: Running
      progress: 0/1
      startedAt: "2022-01-05T17:02:35Z"
      templateName: whalesay
      templateScope: local/steps-qcjw9
      type: Pod
    steps-qcjw9-3301525239:
      boundaryID: steps-qcjw9
      children:
      - steps-qcjw9-2094437448
      displayName: '[0]'
      finishedAt: null
      id: steps-qcjw9-3301525239
      name: steps-qcjw9[0]
      phase: Running
      progress: 0/1
      startedAt: "2022-01-05T17:02:35Z"
      templateScope: local/steps-qcjw9
      type: StepGroup
  phase: Running
  progress: 0/1
  startedAt: "2022-01-05T17:02:35Z"

What Kubernetes provider are you using?
GKE
What executor are you running? Docker/K8SAPI/Kubelet/PNS/Emissary
No Idea, this is my installation steps:
kubectl create namespace argo
kubectl apply -n argo -f https://github.com/argoproj/argo-workflows/releases/download/v3.2.6/install.yaml
curl -sLO https://github.com/argoproj/argo-workflows/releases/download/v3.2.6/argo-linux-amd64.gz ... installing client
argo -n argo submit ./steps.yaml -p message="HelpMeSolveThis"

# Logs from the workflow controller:
kubectl logs -n argo deploy/workflow-controller | grep ${workflow}
time="2022-01-05T16:59:10.089Z" level=info msg="Processing workflow" namespace=default workflow=steps-z4csn
time="2022-01-05T16:59:10.103Z" level=info msg="Updated phase  -> Running" namespace=default workflow=steps-z4csn
time="2022-01-05T16:59:10.104Z" level=info msg="Steps node steps-z4csn initialized Running" namespace=default workflow=steps-z4csn
time="2022-01-05T16:59:10.104Z" level=info msg="StepGroup node steps-z4csn-2699925059 initialized Running" namespace=default workflow=steps-z4csn
time="2022-01-05T16:59:10.105Z" level=info msg="Pod node steps-z4csn-3230619924 initialized Pending" namespace=default workflow=steps-z4csn
time="2022-01-05T16:59:10.177Z" level=info msg="Created pod: steps-z4csn[0].hello1 (steps-z4csn-3230619924)" namespace=default workflow=steps-z4csn
time="2022-01-05T16:59:10.177Z" level=info msg="Workflow step group node steps-z4csn-2699925059 not yet completed" namespace=default workflow=steps-z4csn
time="2022-01-05T16:59:10.177Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=steps-z4csn
time="2022-01-05T16:59:10.177Z" level=info msg=reconcileAgentPod namespace=default workflow=steps-z4csn
time="2022-01-05T16:59:10.239Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=1993 workflow=steps-z4csn
time="2022-01-05T16:59:20.190Z" level=info msg="Processing workflow" namespace=default workflow=steps-z4csn
time="2022-01-05T16:59:20.191Z" level=info msg="Updating node steps-z4csn-3230619924 message: ContainerCreating" namespace=default workflow=steps-z4csn
time="2022-01-05T16:59:20.192Z" level=info msg="Workflow step group node steps-z4csn-2699925059 not yet completed" namespace=default workflow=steps-z4csn
time="2022-01-05T16:59:20.192Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=steps-z4csn
time="2022-01-05T16:59:20.192Z" level=info msg=reconcileAgentPod namespace=default workflow=steps-z4csn
time="2022-01-05T16:59:20.208Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=2139 workflow=steps-z4csn
time="2022-01-05T16:59:30.211Z" level=info msg="Processing workflow" namespace=default workflow=steps-z4csn
time="2022-01-05T16:59:30.213Z" level=info msg="Workflow step group node steps-z4csn-2699925059 not yet completed" namespace=default workflow=steps-z4csn
time="2022-01-05T16:59:30.213Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=steps-z4csn
time="2022-01-05T16:59:30.213Z" level=info msg=reconcileAgentPod namespace=default workflow=steps-z4csn
time="2022-01-05T16:59:52.122Z" level=info msg="Processing workflow" namespace=default workflow=steps-z4csn
time="2022-01-05T16:59:52.122Z" level=info msg="Updating node steps-z4csn-3230619924 status Pending -> Running" namespace=default workflow=steps-z4csn
time="2022-01-05T16:59:52.123Z" level=info msg="Workflow step group node steps-z4csn-2699925059 not yet completed" namespace=default workflow=steps-z4csn
time="2022-01-05T16:59:52.123Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=steps-z4csn
time="2022-01-05T16:59:52.123Z" level=info msg=reconcileAgentPod namespace=default workflow=steps-z4csn
time="2022-01-05T16:59:52.143Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=2384 workflow=steps-z4csn
time="2022-01-05T17:00:02.142Z" level=info msg="Processing workflow" namespace=default workflow=steps-z4csn
time="2022-01-05T17:00:02.143Z" level=info msg="Workflow step group node steps-z4csn-2699925059 not yet completed" namespace=default workflow=steps-z4csn
time="2022-01-05T17:00:02.143Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=steps-z4csn
time="2022-01-05T17:00:02.143Z" level=info msg=reconcileAgentPod namespace=default workflow=steps-z4csn
time="2022-01-05T17:02:35.345Z" level=info msg="Processing workflow" namespace=argo workflow=steps-qcjw9
time="2022-01-05T17:02:35.362Z" level=info msg="Updated phase  -> Running" namespace=argo workflow=steps-qcjw9
time="2022-01-05T17:02:35.363Z" level=info msg="Steps node steps-qcjw9 initialized Running" namespace=argo workflow=steps-qcjw9
time="2022-01-05T17:02:35.364Z" level=info msg="StepGroup node steps-qcjw9-3301525239 initialized Running" namespace=argo workflow=steps-qcjw9
time="2022-01-05T17:02:35.364Z" level=info msg="Pod node steps-qcjw9-2094437448 initialized Pending" namespace=argo workflow=steps-qcjw9
time="2022-01-05T17:02:35.402Z" level=info msg="Created pod: steps-qcjw9[0].hello1 (steps-qcjw9-2094437448)" namespace=argo workflow=steps-qcjw9
time="2022-01-05T17:02:35.402Z" level=info msg="Workflow step group node steps-qcjw9-3301525239 not yet completed" namespace=argo workflow=steps-qcjw9
time="2022-01-05T17:02:35.402Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=steps-qcjw9
time="2022-01-05T17:02:35.403Z" level=info msg=reconcileAgentPod namespace=argo workflow=steps-qcjw9
time="2022-01-05T17:02:35.424Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=3577 workflow=steps-qcjw9
time="2022-01-05T17:02:45.416Z" level=info msg="Processing workflow" namespace=argo workflow=steps-qcjw9
time="2022-01-05T17:02:45.417Z" level=info msg="Updating node steps-qcjw9-2094437448 message: ContainerCreating" namespace=argo workflow=steps-qcjw9
time="2022-01-05T17:02:45.418Z" level=info msg="Workflow step group node steps-qcjw9-3301525239 not yet completed" namespace=argo workflow=steps-qcjw9
time="2022-01-05T17:02:45.418Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=steps-qcjw9
time="2022-01-05T17:02:45.418Z" level=info msg=reconcileAgentPod namespace=argo workflow=steps-qcjw9
time="2022-01-05T17:02:45.443Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=3652 workflow=steps-qcjw9
time="2022-01-05T17:02:55.445Z" level=info msg="Processing workflow" namespace=argo workflow=steps-qcjw9
time="2022-01-05T17:02:55.447Z" level=info msg="Workflow step group node steps-qcjw9-3301525239 not yet completed" namespace=argo workflow=steps-qcjw9
time="2022-01-05T17:02:55.447Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=steps-qcjw9
time="2022-01-05T17:02:55.447Z" level=info msg=reconcileAgentPod namespace=argo workflow=steps-qcjw9
time="2022-01-05T17:03:15.662Z" level=info msg="Processing workflow" namespace=argo workflow=steps-qcjw9
time="2022-01-05T17:03:15.663Z" level=info msg="Updating node steps-qcjw9-2094437448 status Pending -> Running" namespace=argo workflow=steps-qcjw9
time="2022-01-05T17:03:15.665Z" level=info msg="Workflow step group node steps-qcjw9-3301525239 not yet completed" namespace=argo workflow=steps-qcjw9
time="2022-01-05T17:03:15.665Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=steps-qcjw9
time="2022-01-05T17:03:15.665Z" level=info msg=reconcileAgentPod namespace=argo workflow=steps-qcjw9
time="2022-01-05T17:03:15.689Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=3880 workflow=steps-qcjw9
time="2022-01-05T17:03:25.690Z" level=info msg="Processing workflow" namespace=argo workflow=steps-qcjw9
time="2022-01-05T17:03:25.692Z" level=info msg="Workflow step group node steps-qcjw9-3301525239 not yet completed" namespace=argo workflow=steps-qcjw9
time="2022-01-05T17:03:25.692Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=steps-qcjw9
time="2022-01-05T17:03:25.692Z" level=info msg=reconcileAgentPod namespace=argo workflow=steps-qcjw9
time="2022-01-05T17:23:25.691Z" level=info msg="Processing workflow" namespace=argo workflow=steps-qcjw9
time="2022-01-05T17:23:25.693Z" level=info msg="Workflow step group node steps-qcjw9-3301525239 not yet completed" namespace=argo workflow=steps-qcjw9
time="2022-01-05T17:23:25.693Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=steps-qcjw9
time="2022-01-05T17:23:25.693Z" level=info msg=reconcileAgentPod namespace=argo workflow=steps-qcjw9
time="2022-01-05T17:43:25.692Z" level=info msg="Processing workflow" namespace=argo workflow=steps-qcjw9
time="2022-01-05T17:43:25.694Z" level=info msg="Workflow step group node steps-qcjw9-3301525239 not yet completed" namespace=argo workflow=steps-qcjw9
time="2022-01-05T17:43:25.694Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=steps-qcjw9
time="2022-01-05T17:43:25.694Z" level=info msg=reconcileAgentPod namespace=argo workflow=steps-qcjw9

this also might help "logs from wait container"
![image](https://user-images.githubusercontent.com/37670762/148259631-bd3c3cfb-31f8-475c-a9f2-23dbd88cc882.png)



# If the workflow's pods have not been created, you can skip the rest of the diagnostics.

# The workflow's pods that are problematic:
kubectl get pod -o yaml -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

# Logs from in your workflow's wait container, something like:
kubectl logs -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

workflow - argoexec container stuck on wait after main container finishes #7480

{{title}}

Replies: 2 comments 3 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

workflow - argoexec container stuck on wait after main container finishes #7480

m0nk3y-s3c Jan 1, 2022

Replies: 2 comments · 3 replies

terrytangyuan Jan 3, 2022 Maintainer

m0nk3y-s3c Jan 7, 2022 Author

terrytangyuan Jan 7, 2022 Maintainer

m0nk3y-s3c Jan 8, 2022 Author

m0nk3y-s3c Jan 5, 2022 Author

Thanks @terrytangyuan, here it is

Summary

Diagnostics

m0nk3y-s3c
Jan 1, 2022

Replies: 2 comments 3 replies

terrytangyuan
Jan 3, 2022
Maintainer

m0nk3y-s3c Jan 7, 2022
Author

terrytangyuan Jan 7, 2022
Maintainer

m0nk3y-s3c Jan 8, 2022
Author

m0nk3y-s3c
Jan 5, 2022
Author