Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

{{ workflow.failures }} doesn't get a value in lifecycle hook #8882

Closed
3 tasks done
ezk84 opened this issue May 30, 2022 · 8 comments · Fixed by #9009
Closed
3 tasks done

{{ workflow.failures }} doesn't get a value in lifecycle hook #8882

ezk84 opened this issue May 30, 2022 · 8 comments · Fixed by #9009
Labels
area/hooks area/templating Templating with `{{...}}` good first issue Good for newcomers type/bug

Comments

@ezk84
Copy link
Contributor

ezk84 commented May 30, 2022 •
edited by agilgur5
Loading

Checklist

  • Double-checked my configuration.
  • Tested using the latest version.
  • Used the Emissary executor.

Summary

What happened/what you expected to happen?

The message that is output by the hook named on-fail should have the {{ workflow.failures }} replaced by a JSON object as per the documentation.

Instead the literal string {{ workflow.failures }} gets output.

This is also visible when one inspects the INPUTS/OUTPUTS tab in the Workflow Details view in the UI.

What version are you running?

$ argo version
argo: v3.3.6
  BuildDate: 2022-05-26T01:07:24Z
  GitCommit: 2b428be8001a9d5d232dbd52d7e902812107eb28
  GitTreeState: clean
  GitTag: v3.3.6
  GoVersion: go1.17.10
  Compiler: gc
  Platform: windows/amd64

Diagnostics

Paste the smallest workflow that reproduces the bug. We must be able to run the workflow.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  name: failure-hook-fail
  namespace: argo
spec:
  entrypoint: main
  hooks:
    on-fail:
      expression: workflow.status == "Failed" || workflow.status == "Error"
      template: message
      arguments:
        parameters:
          - name: message
            value: |
              Workflow {{ workflow.name }} {{ workflow.status }} {{ workflow.failures }}

  templates:
    - name: main
      dag:
        tasks:
          - name: fail
            inline:
              script:
                image: alpine:latest
                command: [sh]
                source: |
                  set -e
                  non-existent
    - name: message
      inputs:
        parameters:
          - name: message
            default: enter message
      container:
        image: docker/whalesay
        command: [cowsay]
        args:
          - "{{ inputs.parameters.message }}"

Logs from the workflow controller:

kubectl logs -n argo deploy/workflow-controller | grep ${workflow} 
time="2022-05-30T09:08:40.904Z" level=info msg="Processing workflow" namespace=argo workflow=failure-hook-fail
time="2022-05-30T09:08:40.915Z" level=info msg="Updated phase  -> Running" namespace=argo workflow=failure-hook-fail
time="2022-05-30T09:08:40.915Z" level=info msg="DAG node failure-hook-fail initialized Running" namespace=argo workflow=failure-hook-fail
time="2022-05-30T09:08:40.916Z" level=info msg="All of node failure-hook-fail.fail dependencies [] completed" namespace=argo workflow=failure-hook-fail
time="2022-05-30T09:08:40.916Z" level=info msg="Pod node failure-hook-fail-3581964464 initialized Pending" namespace=argo workflow=failure-hook-fail
time="2022-05-30T09:08:40.952Z" level=info msg="Created pod: failure-hook-fail.fail (failure-hook-fail-3581964464)" namespace=argo workflow=failure-hook-fail
time="2022-05-30T09:08:40.952Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=failure-hook-fail
time="2022-05-30T09:08:40.952Z" level=info msg=reconcileAgentPod namespace=argo workflow=failure-hook-fail
time="2022-05-30T09:08:40.961Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=373271431 workflow=failure-hook-fail
time="2022-05-30T09:08:47.187Z" level=info msg="Processing workflow" namespace=argo workflow=failure-hook-fail
time="2022-05-30T09:08:47.188Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=failure-hook-fail
time="2022-05-30T09:08:47.188Z" level=info msg="node changed" new.message="Error (exit code 127)" new.phase=Failed new.progress=0/1 nodeID=failure-hook-fail-3581964464 old.message= old.phase=Pending old.progress=0/1
time="2022-05-30T09:08:47.189Z" level=info msg="Outbound nodes of failure-hook-fail set to [failure-hook-fail-3581964464]" namespace=argo workflow=failure-hook-fail
time="2022-05-30T09:08:47.189Z" level=info msg="node failure-hook-fail phase Running -> Failed" namespace=argo workflow=failure-hook-fail
time="2022-05-30T09:08:47.189Z" level=info msg="node failure-hook-fail finished: 2022-05-30 09:08:47.189153813 +0000 UTC" namespace=argo workflow=failure-hook-fail
time="2022-05-30T09:08:47.189Z" level=info msg="Checking daemoned children of failure-hook-fail" namespace=argo workflow=failure-hook-fail
time="2022-05-30T09:08:47.189Z" level=info msg="Running workflow level hooks" lifeCycleHook=on-fail namespace=argo node=failure-hook-fail.hooks.on-fail workflow=failure-hook-fail
time="2022-05-30T09:08:47.189Z" level=info msg="Pod node failure-hook-fail-3397319682 initialized Pending" namespace=argo workflow=failure-hook-fail
time="2022-05-30T09:08:47.223Z" level=info msg="Created pod: failure-hook-fail.hooks.on-fail (failure-hook-fail-3397319682)" namespace=argo workflow=failure-hook-fail
time="2022-05-30T09:08:47.223Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=failure-hook-fail
time="2022-05-30T09:08:47.223Z" level=info msg=reconcileAgentPod namespace=argo workflow=failure-hook-fail
time="2022-05-30T09:08:47.223Z" level=info msg="Updated phase Running -> Failed" namespace=argo workflow=failure-hook-fail
time="2022-05-30T09:08:47.223Z" level=info msg="Marking workflow completed" namespace=argo workflow=failure-hook-fail
time="2022-05-30T09:08:47.223Z" level=info msg="Doesn't match with archive label selector. Skipping Archive" namespace=argo workflow=failure-hook-fail
time="2022-05-30T09:08:47.223Z" level=info msg="Checking daemoned children of " namespace=argo workflow=failure-hook-fail
time="2022-05-30T09:08:47.228Z" level=info msg="cleaning up pod" action=deletePod key=argo/failure-hook-fail-1340600742-agent/deletePod
time="2022-05-30T09:08:47.248Z" level=info msg="Workflow update successful" namespace=argo phase=Failed resourceVersion=373271480 workflow=failure-hook-fail
time="2022-05-30T09:08:47.268Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/failure-hook-fail-3581964464/labelPodCompleted

The workflow's pods that are problematic:

Pods: 
apiVersion: v1
items:
- apiVersion: v1
  kind: Pod
  metadata:
    annotations:
      kubectl.kubernetes.io/default-container: main
      workflows.argoproj.io/node-id: failure-hook-fail-3397319682
      workflows.argoproj.io/node-name: failure-hook-fail.hooks.on-fail
    creationTimestamp: "2022-05-30T09:08:47Z"
    labels:
      workflows.argoproj.io/completed: "false"
      workflows.argoproj.io/workflow: failure-hook-fail
    name: failure-hook-fail-3397319682
    namespace: argo
    ownerReferences:
    - apiVersion: argoproj.io/v1alpha1
      blockOwnerDeletion: true
      controller: true
      kind: Workflow
      name: failure-hook-fail
      uid: 5d142222-8e97-4e77-bb77-a717b5c1bcc1
    resourceVersion: "373271519"
    uid: cc76e0c1-d055-4710-80e5-10de38ac213c
  spec:
    containers:
    - command:
      - argoexec
      - wait
      - --loglevel
      - info
      env:
      - name: ARGO_POD_NAME
        valueFrom:
          fieldRef:
            apiVersion: v1
            fieldPath: metadata.name
      - name: ARGO_POD_UID
        valueFrom:
          fieldRef:
            apiVersion: v1
            fieldPath: metadata.uid
      - name: ARGO_CONTAINER_RUNTIME_EXECUTOR
        value: emissary
      - name: GODEBUG
        value: x509ignoreCN=0
      - name: ARGO_WORKFLOW_NAME
        value: failure-hook-fail
      - name: ARGO_CONTAINER_NAME
        value: wait
      - name: ARGO_TEMPLATE
        value: '{"name":"message","inputs":{"parameters":[{"name":"message","default":"enter
          message","value":"Workflow failure-hook-fail  {{ workflow.failures }}\n"}]},"outputs":{},"metadata":{},"container":{"name":"","image":"docker/whalesay","command":["cowsay"],"args":["Workflow
          failure-hook-fail  {{ workflow.failures }}\n"],"resources":{}}}'
      - name: ARGO_NODE_ID
        value: failure-hook-fail-3397319682
      - name: ARGO_INCLUDE_SCRIPT_OUTPUT
        value: "false"
      - name: ARGO_DEADLINE
        value: "0001-01-01T00:00:00Z"
      - name: ARGO_PROGRESS_FILE
        value: /var/run/argo/progress
      - name: ARGO_PROGRESS_PATCH_TICK_DURATION
        value: 1m0s
      - name: ARGO_PROGRESS_FILE_TICK_DURATION
        value: 3s
      image: quay.io/argoproj/argoexec:v3.3.6
      imagePullPolicy: IfNotPresent
      name: wait
      resources: {}
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
      - mountPath: /var/run/argo
        name: var-run-argo
      - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        name: kube-api-access-49kt9
        readOnly: true
    - args:
      - |
        Workflow failure-hook-fail  {{ workflow.failures }}
      command:
      - /var/run/argo/argoexec
      - emissary
      - --
      - cowsay
      env:
      - name: ARGO_CONTAINER_NAME
        value: main
      - name: ARGO_TEMPLATE
        value: '{"name":"message","inputs":{"parameters":[{"name":"message","default":"enter
          message","value":"Workflow failure-hook-fail  {{ workflow.failures }}\n"}]},"outputs":{},"metadata":{},"container":{"name":"","image":"docker/whalesay","command":["cowsay"],"args":["Workflow
          failure-hook-fail  {{ workflow.failures }}\n"],"resources":{}}}'
      - name: ARGO_NODE_ID
        value: failure-hook-fail-3397319682
      - name: ARGO_INCLUDE_SCRIPT_OUTPUT
        value: "false"
      - name: ARGO_DEADLINE
        value: "0001-01-01T00:00:00Z"
      - name: ARGO_PROGRESS_FILE
        value: /var/run/argo/progress
      - name: ARGO_PROGRESS_PATCH_TICK_DURATION
        value: 1m0s
      - name: ARGO_PROGRESS_FILE_TICK_DURATION
        value: 3s
      image: docker/whalesay
      imagePullPolicy: Always
      name: main
      resources: {}
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
      - mountPath: /var/run/argo
        name: var-run-argo
      - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        name: kube-api-access-49kt9
        readOnly: true
    dnsPolicy: ClusterFirst
    enableServiceLinks: true
    initContainers:
    - command:
      - argoexec
      - init
      - --loglevel
      - info
      env:
      - name: ARGO_POD_NAME
        valueFrom:
          fieldRef:
            apiVersion: v1
            fieldPath: metadata.name
      - name: ARGO_POD_UID
        valueFrom:
          fieldRef:
            apiVersion: v1
            fieldPath: metadata.uid
      - name: ARGO_CONTAINER_RUNTIME_EXECUTOR
        value: emissary
      - name: GODEBUG
        value: x509ignoreCN=0
      - name: ARGO_WORKFLOW_NAME
        value: failure-hook-fail
      - name: ARGO_CONTAINER_NAME
        value: init
      - name: ARGO_TEMPLATE
        value: '{"name":"message","inputs":{"parameters":[{"name":"message","default":"enter
          message","value":"Workflow failure-hook-fail  {{ workflow.failures }}\n"}]},"outputs":{},"metadata":{},"container":{"name":"","image":"docker/whalesay","command":["cowsay"],"args":["Workflow
          failure-hook-fail  {{ workflow.failures }}\n"],"resources":{}}}'
      - name: ARGO_NODE_ID
        value: failure-hook-fail-3397319682
      - name: ARGO_INCLUDE_SCRIPT_OUTPUT
        value: "false"
      - name: ARGO_DEADLINE
        value: "0001-01-01T00:00:00Z"
      - name: ARGO_PROGRESS_FILE
        value: /var/run/argo/progress
      - name: ARGO_PROGRESS_PATCH_TICK_DURATION
        value: 1m0s
      - name: ARGO_PROGRESS_FILE_TICK_DURATION
        value: 3s
      image: quay.io/argoproj/argoexec:v3.3.6
      imagePullPolicy: IfNotPresent
      name: init
      resources: {}
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
      - mountPath: /var/run/argo
        name: var-run-argo
      - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        name: kube-api-access-49kt9
        readOnly: true
    nodeName: gke-hms-1-n2-pool-a7c7446b-pey9
    preemptionPolicy: PreemptLowerPriority
    priority: 0
    restartPolicy: Never
    schedulerName: default-scheduler
    securityContext: {}
    serviceAccount: default
    serviceAccountName: default
    terminationGracePeriodSeconds: 30
    tolerations:
    - effect: NoExecute
      key: node.kubernetes.io/not-ready
      operator: Exists
      tolerationSeconds: 300
    - effect: NoExecute
      key: node.kubernetes.io/unreachable
      operator: Exists
      tolerationSeconds: 300
    volumes:
    - emptyDir: {}
      name: var-run-argo
    - name: kube-api-access-49kt9
      projected:
        defaultMode: 420
        sources:
        - serviceAccountToken:
            expirationSeconds: 3607
            path: token
        - configMap:
            items:
            - key: ca.crt
              path: ca.crt
            name: kube-root-ca.crt
        - downwardAPI:
            items:
            - fieldRef:
                apiVersion: v1
                fieldPath: metadata.namespace
              path: namespace
  status:
    conditions:
    - lastProbeTime: null
      lastTransitionTime: "2022-05-30T09:08:49Z"
      reason: PodCompleted
      status: "True"
      type: Initialized
    - lastProbeTime: null
      lastTransitionTime: "2022-05-30T09:08:47Z"
      reason: PodCompleted
      status: "False"
      type: Ready
    - lastProbeTime: null
      lastTransitionTime: "2022-05-30T09:08:47Z"
      reason: PodCompleted
      status: "False"
      type: ContainersReady
    - lastProbeTime: null
      lastTransitionTime: "2022-05-30T09:08:47Z"
      status: "True"
      type: PodScheduled
    containerStatuses:
    - containerID: docker://4b67b9728911f4e0f155b98ecd61fc06487f59823e0308aa2f49ac3c0f1e0286
      image: docker/whalesay:latest
      imageID: docker-pullable://docker/whalesay@sha256:178598e51a26abbc958b8a2e48825c90bc22e641de3d31e18aaf55f3258ba93b
      lastState: {}
      name: main
      ready: false
      restartCount: 0
      started: false
      state:
        terminated:
          containerID: docker://4b67b9728911f4e0f155b98ecd61fc06487f59823e0308aa2f49ac3c0f1e0286
          exitCode: 0
          finishedAt: "2022-05-30T09:08:51Z"
          reason: Completed
          startedAt: "2022-05-30T09:08:51Z"
    - containerID: docker://5b416debcfa9d038b728cb852caf753269f687d80363c2f523ffb6f4380a18ab
      image: quay.io/argoproj/argoexec:v3.3.6
      imageID: docker-pullable://quay.io/argoproj/argoexec@sha256:51764b8af2b2ec73d91b46333182e5bb48ab19455ce67e4d76846515a1252a19
      lastState: {}
      name: wait
      ready: false
      restartCount: 0
      started: false
      state:
        terminated:
          containerID: docker://5b416debcfa9d038b728cb852caf753269f687d80363c2f523ffb6f4380a18ab
          exitCode: 0
          finishedAt: "2022-05-30T09:08:51Z"
          reason: Completed
          startedAt: "2022-05-30T09:08:49Z"
    hostIP: 10.152.0.27
    initContainerStatuses:
    - containerID: docker://0dc1a8596824d1fe1e5b95a7510e4981737eb3af2c2458e46c21fae542c17c3e
      image: quay.io/argoproj/argoexec:v3.3.6
      imageID: docker-pullable://quay.io/argoproj/argoexec@sha256:51764b8af2b2ec73d91b46333182e5bb48ab19455ce67e4d76846515a1252a19
      lastState: {}
      name: init
      ready: true
      restartCount: 0
      state:
        terminated:
          containerID: docker://0dc1a8596824d1fe1e5b95a7510e4981737eb3af2c2458e46c21fae542c17c3e
          exitCode: 0
          finishedAt: "2022-05-30T09:08:48Z"
          reason: Completed
          startedAt: "2022-05-30T09:08:48Z"
    phase: Succeeded
    podIP: 10.16.0.122
    podIPs:
    - ip: 10.16.0.122
    qosClass: BestEffort
    startTime: "2022-05-30T09:08:47Z"
- apiVersion: v1
  kind: Pod
  metadata:
    annotations:
      kubectl.kubernetes.io/default-container: main
      workflows.argoproj.io/node-id: failure-hook-fail-3581964464
      workflows.argoproj.io/node-name: failure-hook-fail.fail
    creationTimestamp: "2022-05-30T09:08:40Z"
    labels:
      workflows.argoproj.io/completed: "true"
      workflows.argoproj.io/workflow: failure-hook-fail
    name: failure-hook-fail-3581964464
    namespace: argo
    ownerReferences:
    - apiVersion: argoproj.io/v1alpha1
      blockOwnerDeletion: true
      controller: true
      kind: Workflow
      name: failure-hook-fail
      uid: 5d142222-8e97-4e77-bb77-a717b5c1bcc1
    resourceVersion: "373271482"
    uid: 0f3781a5-a3a5-4fe8-842c-6ed5dc54fcef
  spec:
    containers:
    - command:
      - argoexec
      - wait
      - --loglevel
      - info
      env:
      - name: ARGO_POD_NAME
        valueFrom:
          fieldRef:
            apiVersion: v1
            fieldPath: metadata.name
      - name: ARGO_POD_UID
        valueFrom:
          fieldRef:
            apiVersion: v1
            fieldPath: metadata.uid
      - name: ARGO_CONTAINER_RUNTIME_EXECUTOR
        value: emissary
      - name: GODEBUG
        value: x509ignoreCN=0
      - name: ARGO_WORKFLOW_NAME
        value: failure-hook-fail
      - name: ARGO_CONTAINER_NAME
        value: wait
      - name: ARGO_TEMPLATE
        value: '{"inputs":{},"outputs":{},"metadata":{},"script":{"name":"","image":"alpine:latest","command":["sh"],"resources":{},"source":"set
          -e\nnon-existent\n"}}'
      - name: ARGO_NODE_ID
        value: failure-hook-fail-3581964464
      - name: ARGO_INCLUDE_SCRIPT_OUTPUT
        value: "false"
      - name: ARGO_DEADLINE
        value: "0001-01-01T00:00:00Z"
      - name: ARGO_PROGRESS_FILE
        value: /var/run/argo/progress
      - name: ARGO_PROGRESS_PATCH_TICK_DURATION
        value: 1m0s
      - name: ARGO_PROGRESS_FILE_TICK_DURATION
        value: 3s
      image: quay.io/argoproj/argoexec:v3.3.6
      imagePullPolicy: IfNotPresent
      name: wait
      resources: {}
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
      - mountPath: /mainctrfs/argo/staging
        name: argo-staging
      - mountPath: /var/run/argo
        name: var-run-argo
      - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        name: kube-api-access-bxwqk
        readOnly: true
    - args:
      - /argo/staging/script
      command:
      - /var/run/argo/argoexec
      - emissary
      - --
      - sh
      env:
      - name: ARGO_CONTAINER_NAME
        value: main
      - name: ARGO_TEMPLATE
        value: '{"inputs":{},"outputs":{},"metadata":{},"script":{"name":"","image":"alpine:latest","command":["sh"],"resources":{},"source":"set
          -e\nnon-existent\n"}}'
      - name: ARGO_NODE_ID
        value: failure-hook-fail-3581964464
      - name: ARGO_INCLUDE_SCRIPT_OUTPUT
        value: "false"
      - name: ARGO_DEADLINE
        value: "0001-01-01T00:00:00Z"
      - name: ARGO_PROGRESS_FILE
        value: /var/run/argo/progress
      - name: ARGO_PROGRESS_PATCH_TICK_DURATION
        value: 1m0s
      - name: ARGO_PROGRESS_FILE_TICK_DURATION
        value: 3s
      image: alpine:latest
      imagePullPolicy: Always
      name: main
      resources: {}
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
      - mountPath: /argo/staging
        name: argo-staging
      - mountPath: /var/run/argo
        name: var-run-argo
      - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        name: kube-api-access-bxwqk
        readOnly: true
    dnsPolicy: ClusterFirst
    enableServiceLinks: true
    initContainers:
    - command:
      - argoexec
      - init
      - --loglevel
      - info
      env:
      - name: ARGO_POD_NAME
        valueFrom:
          fieldRef:
            apiVersion: v1
            fieldPath: metadata.name
      - name: ARGO_POD_UID
        valueFrom:
          fieldRef:
            apiVersion: v1
            fieldPath: metadata.uid
      - name: ARGO_CONTAINER_RUNTIME_EXECUTOR
        value: emissary
      - name: GODEBUG
        value: x509ignoreCN=0
      - name: ARGO_WORKFLOW_NAME
        value: failure-hook-fail
      - name: ARGO_CONTAINER_NAME
        value: init
      - name: ARGO_TEMPLATE
        value: '{"inputs":{},"outputs":{},"metadata":{},"script":{"name":"","image":"alpine:latest","command":["sh"],"resources":{},"source":"set
          -e\nnon-existent\n"}}'
      - name: ARGO_NODE_ID
        value: failure-hook-fail-3581964464
      - name: ARGO_INCLUDE_SCRIPT_OUTPUT
        value: "false"
      - name: ARGO_DEADLINE
        value: "0001-01-01T00:00:00Z"
      - name: ARGO_PROGRESS_FILE
        value: /var/run/argo/progress
      - name: ARGO_PROGRESS_PATCH_TICK_DURATION
        value: 1m0s
      - name: ARGO_PROGRESS_FILE_TICK_DURATION
        value: 3s
      image: quay.io/argoproj/argoexec:v3.3.6
      imagePullPolicy: IfNotPresent
      name: init
      resources: {}
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
      - mountPath: /argo/staging
        name: argo-staging
      - mountPath: /var/run/argo
        name: var-run-argo
      - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        name: kube-api-access-bxwqk
        readOnly: true
    nodeName: gke-hms-1-n2-pool-a7c7446b-pey9
    preemptionPolicy: PreemptLowerPriority
    priority: 0
    restartPolicy: Never
    schedulerName: default-scheduler
    securityContext: {}
    serviceAccount: default
    serviceAccountName: default
    terminationGracePeriodSeconds: 30
    tolerations:
    - effect: NoExecute
      key: node.kubernetes.io/not-ready
      operator: Exists
      tolerationSeconds: 300
    - effect: NoExecute
      key: node.kubernetes.io/unreachable
      operator: Exists
      tolerationSeconds: 300
    volumes:
    - emptyDir: {}
      name: var-run-argo
    - emptyDir: {}
      name: argo-staging
    - name: kube-api-access-bxwqk
      projected:
        defaultMode: 420
        sources:
        - serviceAccountToken:
            expirationSeconds: 3607
            path: token
        - configMap:
            items:
            - key: ca.crt
              path: ca.crt
            name: kube-root-ca.crt
        - downwardAPI:
            items:
            - fieldRef:
                apiVersion: v1
                fieldPath: metadata.namespace
              path: namespace
  status:
    conditions:
    - lastProbeTime: null
      lastTransitionTime: "2022-05-30T09:08:43Z"
      status: "True"
      type: Initialized
    - lastProbeTime: null
      lastTransitionTime: "2022-05-30T09:08:41Z"
      message: 'containers with unready status: [wait main]'
      reason: ContainersNotReady
      status: "False"
      type: Ready
    - lastProbeTime: null
      lastTransitionTime: "2022-05-30T09:08:41Z"
      message: 'containers with unready status: [wait main]'
      reason: ContainersNotReady
      status: "False"
      type: ContainersReady
    - lastProbeTime: null
      lastTransitionTime: "2022-05-30T09:08:40Z"
      status: "True"
      type: PodScheduled
    containerStatuses:
    - containerID: docker://999c0da6249ad23058b037086e7f370a48cedf10556ed27a0065cbea72d36e88
      image: alpine:latest
      imageID: docker-pullable://alpine@sha256:686d8c9dfa6f3ccfc8230bc3178d23f84eeaf7e457f36f271ab1acc53015037c
      lastState: {}
      name: main
      ready: false
      restartCount: 0
      started: false
      state:
        terminated:
          containerID: docker://999c0da6249ad23058b037086e7f370a48cedf10556ed27a0065cbea72d36e88
          exitCode: 127
          finishedAt: "2022-05-30T09:08:44Z"
          reason: Error
          startedAt: "2022-05-30T09:08:44Z"
    - containerID: docker://bc6820969be4b9505e19990d01451fefa18b3cd0b1e5e9c59c69df51c4a085ca
      image: quay.io/argoproj/argoexec:v3.3.6
      imageID: docker-pullable://quay.io/argoproj/argoexec@sha256:51764b8af2b2ec73d91b46333182e5bb48ab19455ce67e4d76846515a1252a19
      lastState: {}
      name: wait
      ready: false
      restartCount: 0
      started: false
      state:
        terminated:
          containerID: docker://bc6820969be4b9505e19990d01451fefa18b3cd0b1e5e9c59c69df51c4a085ca
          exitCode: 0
          finishedAt: "2022-05-30T09:08:45Z"
          reason: Completed
          startedAt: "2022-05-30T09:08:43Z"
    hostIP: 10.152.0.27
    initContainerStatuses:
    - containerID: docker://1782031c875b4e9f99c312f2e82e813917d1a389a753ee90c3afa8dd21368a9c
      image: quay.io/argoproj/argoexec:v3.3.6
      imageID: docker-pullable://quay.io/argoproj/argoexec@sha256:51764b8af2b2ec73d91b46333182e5bb48ab19455ce67e4d76846515a1252a19
      lastState: {}
      name: init
      ready: true
      restartCount: 0
      state:
        terminated:
          containerID: docker://1782031c875b4e9f99c312f2e82e813917d1a389a753ee90c3afa8dd21368a9c
          exitCode: 0
          finishedAt: "2022-05-30T09:08:42Z"
          reason: Completed
          startedAt: "2022-05-30T09:08:42Z"
    phase: Failed
    podIP: 10.16.0.121
    podIPs:
    - ip: 10.16.0.121
    qosClass: BestEffort
    startTime: "2022-05-30T09:08:41Z"
kind: List
metadata:
  resourceVersion: ""

Logs from in your workflow's wait container:

time="2022-05-30T09:08:49.720Z" level=info msg="Creating a emissary executor"
time="2022-05-30T09:08:49.720Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2022-05-30T09:08:49.721Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=argo podName=failure-hook-fail-3397319682 template="{\"name\":\"message\",\"inputs\":{\"parameters\":[{\"name\":\"message\",\"default\":\"enter message\",\"value\":\"Workflow failure-hook-fail  {{ workflow.failures }}\\n\"}]},\"outputs\":{},\"metadata\":{},\"container\":{\"name\":\"\",\"image\":\"docker/whalesay\",\"command\":[\"cowsay\"],\"args\":[\"Workflow failure-hook-fail  {{ workflow.failures }}\\n\"],\"resources\":{}}}" version="&Version{Version:v3.3.6,BuildDate:2022-05-25T23:47:50Z,GitCommit:2b428be8001a9d5d232dbd52d7e902812107eb28,GitTag:v3.3.6,GitTreeState:clean,GoVersion:go1.17.10,Compiler:gc,Platform:linux/amd64,}"
time="2022-05-30T09:08:49.721Z" level=info msg="Starting deadline monitor"
time="2022-05-30T09:08:51.721Z" level=info msg="Main container completed"
time="2022-05-30T09:08:51.721Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2022-05-30T09:08:51.721Z" level=info msg="No output parameters"
time="2022-05-30T09:08:51.721Z" level=info msg="No output artifacts"
time="2022-05-30T09:08:51.721Z" level=info msg="Killing sidecars []"
time="2022-05-30T09:08:51.721Z" level=info msg="Alloc=5560 TotalAlloc=10409 Sys=18130 NumGC=3 Goroutines=7"
time="2022-05-30T09:08:43.720Z" level=info msg="Creating a emissary executor"
time="2022-05-30T09:08:43.720Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2022-05-30T09:08:43.720Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=argo podName=failure-hook-fail-3581964464 template="{\"inputs\":{},\"outputs\":{},\"metadata\":{},\"script\":{\"name\":\"\",\"image\":\"alpine:latest\",\"command\":[\"sh\"],\"resources\":{},\"source\":\"set -e\\nnon-existent\\n\"}}" version="&Version{Version:v3.3.6,BuildDate:2022-05-25T23:47:50Z,GitCommit:2b428be8001a9d5d232dbd52d7e902812107eb28,GitTag:v3.3.6,GitTreeState:clean,GoVersion:go1.17.10,Compiler:gc,Platform:linux/amd64,}"
time="2022-05-30T09:08:43.720Z" level=info msg="Starting deadline monitor"
time="2022-05-30T09:08:45.722Z" level=info msg="Main container completed"
time="2022-05-30T09:08:45.722Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2022-05-30T09:08:45.722Z" level=info msg="No output parameters"
time="2022-05-30T09:08:45.722Z" level=info msg="No output artifacts"
time="2022-05-30T09:08:45.722Z" level=info msg="Killing sidecars []"
time="2022-05-30T09:08:45.722Z" level=info msg="Alloc=5569 TotalAlloc=10411 Sys=18130 NumGC=3 Goroutines=7"

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.
@ezk84
Copy link
Contributor Author

ezk84 commented May 30, 2022 •
edited
Loading

Looking at the doco, it's made clear that the workflow.failures variable is available for onExit handlers, and not necessarily for lifecycle hooks running with the condition workflow.status == "Failed". Thus this is an enhancement. Can't edit the tags though :/

@sandeepk8s
Copy link
Contributor

sandeepk8s commented May 30, 2022 •
edited
Loading

@ezk84 facing the similar issue! 😅 Glad you opened an issue! I was about to! {{workflow.failures}} was not holding any data.

Also there are few issues which I observed using lifecycle hooks and HTTP templates.

@sarabala1979
Copy link
Member

sarabala1979 commented May 30, 2022 •
edited
Loading

It is a bug workflow.failures is not included when calling the lifecycle hook.

argo-workflows/workflow/controller/operator.go

Lines 389 to 409 in 093a6fe

var failures []failedNodeStatus
for _, node := range woc.wf.Status.Nodes {
if node.Phase == wfv1.NodeFailed || node.Phase == wfv1.NodeError {
failures = append(failures,
failedNodeStatus{
DisplayName: node.DisplayName,
Message: node.Message,
TemplateName: node.TemplateName,
Phase: string(node.Phase),
PodName: node.ID,
FinishedAt: node.FinishedAt,
})
}
}
failedNodeBytes, err := json.Marshal(failures)
if err != nil {
woc.log.Errorf("Error marshalling failed nodes list: %+v", err)
// No need to return here
}
// This strconv.Quote is necessary so that the escaped quotes are not removed during parameter substitution
woc.globalParams[common.GlobalVarWorkflowFailures] = strconv.Quote(string(failedNodeBytes))

This block of code needs to move above

argo-workflows/workflow/controller/operator.go

Line 375 in 093a6fe

err = woc.executeWfLifeCycleHook(ctx, tmplCtx)

you can convert the block of code as function and call.

Do you like to submit the PR for this?

@sarabala1979 sarabala1979 added the good first issue Good for newcomers label May 31, 2022
@sarabala1979
Copy link
Member

@ezk84 are you planing to submit the PR for this issue?

@ezk84
Copy link
Contributor Author

ezk84 commented May 31, 2022 •
edited
Loading

@sarabala1979 I will attempt it. Tried to get a dev environment running, though I run into problems with the vscode devcontainer.

I imagine I would have to write at least a test to validate this, no?

@sarabala1979
Copy link
Member

need test. Let me know if you have any problem setting up local env

@stale
Copy link

stale bot commented Jun 18, 2022

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If this is a mentoring request, please provide an update here. Thank you for your contributions.

@stale stale bot added the problem/stale This has not had a response in some time label Jun 18, 2022
ezk84 added a commit to ezk84/argo-workflows that referenced this issue Jun 20, 2022
@ezk84
fix: Add workflow failures before hooks run. Fixes argoproj#8882
b6b9bce 
Signed-off-by: Ezequiel Muns <ezeq.au@gmail.com>
ezk84 added a commit to ezk84/argo-workflows that referenced this issue Jun 20, 2022
@ezk84
fix: Add workflow failures before hooks run. Fixes argoproj#8882
64368fb 
Signed-off-by: Ezequiel Muns <ezeq.au@gmail.com>
@ezk84 ezk84 mentioned this issue Jun 20, 2022
Merged
sarabala1979 pushed a commit that referenced this issue Jun 29, 2022
@ezk84
fix: Add workflow failures before hooks run. Fixes #8882 (#9009)
61211f9 
Signed-off-by: Ezequiel Muns <ezeq.au@gmail.com>
@ezk84
Copy link
Contributor Author

ezk84 commented Jul 1, 2022

🕺 Great!

@sarabala1979 sarabala1979 mentioned this issue Jul 30, 2022
Closed
51 tasks
LoricAndre pushed a commit to padoa/argo-workflows that referenced this issue Aug 1, 2022
@ezk84
fix: Add workflow failures before hooks run. Fixes argoproj#8882 (arg…
28e1d44 
…oproj#9009)

Signed-off-by: Ezequiel Muns <ezeq.au@gmail.com>
Signed-off-by: Loric ANDRE <loric.a@padoa-group.com>
@sarabala1979 sarabala1979 mentioned this issue Nov 8, 2022
Closed
reddymh pushed a commit to reddymh/argo-workflows that referenced this issue Jan 2, 2023
@ezk84
fix: Add workflow failures before hooks run. Fixes argoproj#8882 (arg…
efcd305 
…oproj#9009)

Signed-off-by: Ezequiel Muns <ezeq.au@gmail.com>
Signed-off-by: Reddy <Rajshekar.Reddy@lowes.com>
@agilgur5 agilgur5 added area/hooks area/templating Templating with `{{...}}` and removed problem/stale This has not had a response in some time labels Feb 8, 2024
@agilgur5 agilgur5 changed the title {{ workflow.failures }} doesn't get a value in lifecycle hook {{ workflow.failures }} doesn't get a value in lifecycle hook Feb 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/hooks area/templating Templating with `{{...}}` good first issue Good for newcomers type/bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: Add workflow failures before hooks run. Fixes #8882 ezk84/argo-workflows
4 participants
@ezk84 @agilgur5 @sandeepk8s @sarabala1979