Argo-Exec V3.3.9 is recognising a failure exit code as exitCode 0 #9415
Labels
Comments
HRusby
added
type/bug
type/regression
|
@HRusby I can see your main container is exit with exit code=0. |
@sarabala1979 That is correct, however running the container locally results in a non-zero exit code, as per my summary the issue here is that argo is recognising a zero exit code when the image is outputting a non-zero exit code. I'll be uploading a docker image that displays this behaviour this evening (I'm currently blocked from doing so due to a proxy network). I appreciate it may be difficult to understand the problem before getting hands on the image. I should have included this in the original message, though for reference, running the exact same workflow and same version of the container in v3.3.8 results in these logs. Note the exit code in this case is 137. Comparing the results with the logs from v3.3.9 the only differences I can see (excluding guids) is the version of the argoexec image Problematic Pods for V3.3.8apiVersion: v1
items:
- apiVersion: v1
kind: Pod
metadata:
annotations:
kubectl.kubernetes.io/default-container: main
workflows.argoproj.io/node-id: dotnet-exception-test-bslqf
workflows.argoproj.io/node-name: dotnet-exception-test-bslqf
workflows.argoproj.io/outputs: '{"artifacts":[{"name":"main-logs","s3":{"key":"dotnet-exception-test-bslqf/dotnet-exception-test-bslqf/main.log"}}]}'
creationTimestamp: "2022-08-23T18:01:44Z"
labels:
workflows.argoproj.io/completed: "true"
workflows.argoproj.io/workflow: dotnet-exception-test-bslqf
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
f:workflows.argoproj.io/outputs: {}
manager: argoexec
operation: Update
time: "2022-08-23T18:01:48Z"
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:status:
f:conditions:
k:{"type":"ContainersReady"}:
.: {}
f:lastProbeTime: {}
f:lastTransitionTime: {}
f:message: {}
f:reason: {}
f:status: {}
f:type: {}
k:{"type":"Initialized"}:
.: {}
f:lastProbeTime: {}
f:lastTransitionTime: {}
f:status: {}
f:type: {}
k:{"type":"Ready"}:
.: {}
f:lastProbeTime: {}
f:lastTransitionTime: {}
f:message: {}
f:reason: {}
f:status: {}
f:type: {}
f:containerStatuses: {}
f:hostIP: {}
f:initContainerStatuses: {}
f:phase: {}
f:podIP: {}
f:podIPs:
.: {}
k:{"ip":"10.244.12.89"}:
.: {}
f:ip: {}
f:startTime: {}
manager: kubelet
operation: Update
time: "2022-08-23T18:01:50Z"
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:kubectl.kubernetes.io/default-container: {}
f:workflows.argoproj.io/node-id: {}
f:workflows.argoproj.io/node-name: {}
f:labels:
.: {}
f:workflows.argoproj.io/completed: {}
f:workflows.argoproj.io/workflow: {}
f:ownerReferences:
.: {}
k:{"uid":"0bd2b4d9-3e0c-4d83-8ca3-d4555bb2caad"}:
.: {}
f:apiVersion: {}
f:blockOwnerDeletion: {}
f:controller: {}
f:kind: {}
f:name: {}
f:uid: {}
f:spec:
f:containers:
k:{"name":"main"}:
.: {}
f:command: {}
f:env:
.: {}
k:{"name":"ARGO_CONTAINER_NAME"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"ARGO_DEADLINE"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"ARGO_INCLUDE_SCRIPT_OUTPUT"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"ARGO_NODE_ID"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"ARGO_PROGRESS_FILE"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"ARGO_PROGRESS_FILE_TICK_DURATION"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"ARGO_PROGRESS_PATCH_TICK_DURATION"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"ARGO_TEMPLATE"}:
.: {}
f:name: {}
f:value: {}
f:image: {}
f:imagePullPolicy: {}
f:name: {}
f:resources: {}
f:terminationMessagePath: {}
f:terminationMessagePolicy: {}
f:volumeMounts:
.: {}
k:{"mountPath":"/var/run/argo"}:
.: {}
f:mountPath: {}
f:name: {}
k:{"name":"wait"}:
.: {}
f:command: {}
f:env:
.: {}
k:{"name":"ARGO_CONTAINER_NAME"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"ARGO_CONTAINER_RUNTIME_EXECUTOR"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"ARGO_DEADLINE"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"ARGO_INCLUDE_SCRIPT_OUTPUT"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"ARGO_NODE_ID"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"ARGO_POD_NAME"}:
.: {}
f:name: {}
f:valueFrom:
.: {}
f:fieldRef:
.: {}
f:apiVersion: {}
f:fieldPath: {}
k:{"name":"ARGO_POD_UID"}:
.: {}
f:name: {}
f:valueFrom:
.: {}
f:fieldRef:
.: {}
f:apiVersion: {}
f:fieldPath: {}
k:{"name":"ARGO_PROGRESS_FILE"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"ARGO_PROGRESS_FILE_TICK_DURATION"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"ARGO_PROGRESS_PATCH_TICK_DURATION"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"ARGO_TEMPLATE"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"ARGO_WORKFLOW_NAME"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"GODEBUG"}:
.: {}
f:name: {}
f:value: {}
f:image: {}
f:imagePullPolicy: {}
f:name: {}
f:resources: {}
f:terminationMessagePath: {}
f:terminationMessagePolicy: {}
f:volumeMounts:
.: {}
k:{"mountPath":"/argo/secret/argo-minio"}:
.: {}
f:mountPath: {}
f:name: {}
f:readOnly: {}
k:{"mountPath":"/var/run/argo"}:
.: {}
f:mountPath: {}
f:name: {}
f:dnsPolicy: {}
f:enableServiceLinks: {}
f:initContainers:
.: {}
k:{"name":"init"}:
.: {}
f:command: {}
f:env:
.: {}
k:{"name":"ARGO_CONTAINER_NAME"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"ARGO_CONTAINER_RUNTIME_EXECUTOR"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"ARGO_DEADLINE"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"ARGO_INCLUDE_SCRIPT_OUTPUT"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"ARGO_NODE_ID"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"ARGO_POD_NAME"}:
.: {}
f:name: {}
f:valueFrom:
.: {}
f:fieldRef:
.: {}
f:apiVersion: {}
f:fieldPath: {}
k:{"name":"ARGO_POD_UID"}:
.: {}
f:name: {}
f:valueFrom:
.: {}
f:fieldRef:
.: {}
f:apiVersion: {}
f:fieldPath: {}
k:{"name":"ARGO_PROGRESS_FILE"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"ARGO_PROGRESS_FILE_TICK_DURATION"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"ARGO_PROGRESS_PATCH_TICK_DURATION"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"ARGO_TEMPLATE"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"ARGO_WORKFLOW_NAME"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"GODEBUG"}:
.: {}
f:name: {}
f:value: {}
f:image: {}
f:imagePullPolicy: {}
f:name: {}
f:resources: {}
f:terminationMessagePath: {}
f:terminationMessagePolicy: {}
f:volumeMounts:
.: {}
k:{"mountPath":"/argo/secret/argo-minio"}:
.: {}
f:mountPath: {}
f:name: {}
f:readOnly: {}
k:{"mountPath":"/var/run/argo"}:
.: {}
f:mountPath: {}
f:name: {}
f:restartPolicy: {}
f:schedulerName: {}
f:securityContext: {}
f:terminationGracePeriodSeconds: {}
f:volumes:
.: {}
k:{"name":"argo-minio"}:
.: {}
f:name: {}
f:secret:
.: {}
f:defaultMode: {}
f:items: {}
f:secretName: {}
k:{"name":"var-run-argo"}:
.: {}
f:emptyDir: {}
f:name: {}
manager: workflow-controller
operation: Update
time: "2022-08-23T18:01:54Z"
name: dotnet-exception-test-bslqf
namespace: argo
ownerReferences:
- apiVersion: argoproj.io/v1alpha1
blockOwnerDeletion: true
controller: true
kind: Workflow
name: dotnet-exception-test-bslqf
uid: 0bd2b4d9-3e0c-4d83-8ca3-d4555bb2caad
resourceVersion: "363920693"
selfLink: /api/v1/namespaces/argo/pods/dotnet-exception-test-bslqf
uid: e7b60ab6-1489-43fb-8398-2c9790875d18
spec:
containers:
- command:
- argoexec
- wait
- --loglevel
- info
env:
- name: ARGO_POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: ARGO_POD_UID
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.uid
- name: ARGO_CONTAINER_RUNTIME_EXECUTOR
value: emissary
- name: GODEBUG
value: x509ignoreCN=0
- name: ARGO_WORKFLOW_NAME
value: dotnet-exception-test-bslqf
- name: ARGO_CONTAINER_NAME
value: wait
- name: ARGO_TEMPLATE
value: '{"name":"dotnet-exception-test","inputs":{},"outputs":{},"metadata":{},"container":{"name":"","image":"hrusby/dotnet-immediate-exception:1.0.0","command":["dotnet","Immediate-Exception.dll"],"resources":{}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"argo-minio.argo:9000","bucket":"argo-artifacts","insecure":true,"accessKeySecret":{"name":"argo-minio","key":"accesskey"},"secretKeySecret":{"name":"argo-minio","key":"secretkey"},"key":"dotnet-exception-test-bslqf/dotnet-exception-test-bslqf"}}}'
- name: ARGO_NODE_ID
value: dotnet-exception-test-bslqf
- name: ARGO_INCLUDE_SCRIPT_OUTPUT
value: "false"
- name: ARGO_DEADLINE
value: "0001-01-01T00:00:00Z"
- name: ARGO_PROGRESS_FILE
value: /var/run/argo/progress
- name: ARGO_PROGRESS_PATCH_TICK_DURATION
value: 1m0s
- name: ARGO_PROGRESS_FILE_TICK_DURATION
value: 3s
image: quay.io/argoproj/argoexec:v3.3.8
imagePullPolicy: IfNotPresent
name: wait
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /argo/secret/argo-minio
name: argo-minio
readOnly: true
- mountPath: /var/run/argo
name: var-run-argo
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: default-token-4797k
readOnly: true
- command:
- /var/run/argo/argoexec
- emissary
- --
- dotnet
- Immediate-Exception.dll
env:
- name: ARGO_CONTAINER_NAME
value: main
- name: ARGO_TEMPLATE
value: '{"name":"dotnet-exception-test","inputs":{},"outputs":{},"metadata":{},"container":{"name":"","image":"hrusby/dotnet-immediate-exception:1.0.0","command":["dotnet","Immediate-Exception.dll"],"resources":{}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"argo-minio.argo:9000","bucket":"argo-artifacts","insecure":true,"accessKeySecret":{"name":"argo-minio","key":"accesskey"},"secretKeySecret":{"name":"argo-minio","key":"secretkey"},"key":"dotnet-exception-test-bslqf/dotnet-exception-test-bslqf"}}}'
- name: ARGO_NODE_ID
value: dotnet-exception-test-bslqf
- name: ARGO_INCLUDE_SCRIPT_OUTPUT
value: "false"
- name: ARGO_DEADLINE
value: "0001-01-01T00:00:00Z"
- name: ARGO_PROGRESS_FILE
value: /var/run/argo/progress
- name: ARGO_PROGRESS_PATCH_TICK_DURATION
value: 1m0s
- name: ARGO_PROGRESS_FILE_TICK_DURATION
value: 3s
image: hrusby/dotnet-immediate-exception:1.0.0
imagePullPolicy: IfNotPresent
name: main
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/run/argo
name: var-run-argo
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: default-token-4797k
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
initContainers:
- command:
- argoexec
- init
- --loglevel
- info
env:
- name: ARGO_POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: ARGO_POD_UID
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.uid
- name: ARGO_CONTAINER_RUNTIME_EXECUTOR
value: emissary
- name: GODEBUG
value: x509ignoreCN=0
- name: ARGO_WORKFLOW_NAME
value: dotnet-exception-test-bslqf
- name: ARGO_CONTAINER_NAME
value: init
- name: ARGO_TEMPLATE
value: '{"name":"dotnet-exception-test","inputs":{},"outputs":{},"metadata":{},"container":{"name":"","image":"hrusby/dotnet-immediate-exception:1.0.0","command":["dotnet","Immediate-Exception.dll"],"resources":{}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"argo-minio.argo:9000","bucket":"argo-artifacts","insecure":true,"accessKeySecret":{"name":"argo-minio","key":"accesskey"},"secretKeySecret":{"name":"argo-minio","key":"secretkey"},"key":"dotnet-exception-test-bslqf/dotnet-exception-test-bslqf"}}}'
- name: ARGO_NODE_ID
value: dotnet-exception-test-bslqf
- name: ARGO_INCLUDE_SCRIPT_OUTPUT
value: "false"
- name: ARGO_DEADLINE
value: "0001-01-01T00:00:00Z"
- name: ARGO_PROGRESS_FILE
value: /var/run/argo/progress
- name: ARGO_PROGRESS_PATCH_TICK_DURATION
value: 1m0s
- name: ARGO_PROGRESS_FILE_TICK_DURATION
value: 3s
image: quay.io/argoproj/argoexec:v3.3.8
imagePullPolicy: IfNotPresent
name: init
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /argo/secret/argo-minio
name: argo-minio
readOnly: true
- mountPath: /var/run/argo
name: var-run-argo
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: default-token-4797k
readOnly: true
nodeName: lon2kubuat13
preemptionPolicy: PreemptLowerPriority
priority: 0
restartPolicy: Never
schedulerName: default-scheduler
securityContext: {}
serviceAccount: default
serviceAccountName: default
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- emptyDir: {}
name: var-run-argo
- name: argo-minio
secret:
defaultMode: 420
items:
- key: accesskey
path: accesskey
- key: secretkey
path: secretkey
secretName: argo-minio
- name: default-token-4797k
secret:
defaultMode: 420
secretName: default-token-4797k
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2022-08-23T18:01:47Z"
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2022-08-23T18:01:44Z"
message: 'containers with unready status: [wait main]'
reason: ContainersNotReady
status: "False"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2022-08-23T18:01:44Z"
message: 'containers with unready status: [wait main]'
reason: ContainersNotReady
status: "False"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2022-08-23T18:01:44Z"
status: "True"
type: PodScheduled
containerStatuses:
- containerID: docker://a96acb618dd55e2fd5e385d25d7b74790d4508d65e4dae7849a02a2f6cd0c741
image: hrusby/dotnet-immediate-exception:1.0.0
imageID: docker-pullable://hrusby/dotnet-immediate-exception@sha256:e10e4d66a8c35ebd1a917f831bec5d3466292bec1861b758c573057312f087f0
lastState: {}
name: main
ready: false
restartCount: 0
started: false
state:
terminated:
containerID: docker://a96acb618dd55e2fd5e385d25d7b74790d4508d65e4dae7849a02a2f6cd0c741
exitCode: 137
finishedAt: "2022-08-23T18:01:48Z"
reason: Error
startedAt: "2022-08-23T18:01:48Z"
- containerID: docker://5c49b24ce96fdfcf477d90b53e78539468d25fe9bb886914c93355d79b75c3eb
image: quay.io/argoproj/argoexec:v3.3.8
imageID: docker-pullable://quay.io/argoproj/argoexec@sha256:ba2be5594962f60f5a4ece6b16b9fc359280fff983cabef35ac18edcaeb97609
lastState: {}
name: wait
ready: false
restartCount: 0
started: false
state:
terminated:
containerID: docker://5c49b24ce96fdfcf477d90b53e78539468d25fe9bb886914c93355d79b75c3eb
exitCode: 0
finishedAt: "2022-08-23T18:01:48Z"
reason: Completed
startedAt: "2022-08-23T18:01:47Z"
hostIP: 10.50.100.210
initContainerStatuses:
- containerID: docker://a3bc9928df0445037811a966991af5fac51400b53da006dcd8541315bcb4d1e4
image: quay.io/argoproj/argoexec:v3.3.8
imageID: docker-pullable://quay.io/argoproj/argoexec@sha256:ba2be5594962f60f5a4ece6b16b9fc359280fff983cabef35ac18edcaeb97609
lastState: {}
name: init
ready: true
restartCount: 0
state:
terminated:
containerID: docker://a3bc9928df0445037811a966991af5fac51400b53da006dcd8541315bcb4d1e4
exitCode: 0
finishedAt: "2022-08-23T18:01:46Z"
reason: Completed
startedAt: "2022-08-23T18:01:45Z"
phase: Failed
podIP: 10.244.12.89
podIPs:
- ip: 10.244.12.89
qosClass: BestEffort
startTime: "2022-08-23T18:01:44Z"
kind: List
metadata:
resourceVersion: ""
selfLink: "" |
|
@HRusby Argo-Exec won't do anything for the main container's exit-code. if the container exit with a different code, k8s will update on pod spec. I tried below workflow and not able to reproduce it apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: retry-on-error-
spec:
entrypoint: error-container
templates:
- name: error-container
container:
image: python
command: ["python", "-c"]
# fail with a 80% probability
args: ["import random; import sys; exit_code = random.choice(range(0, 5)); sys.exit(137)"]pod spec |
|
@sarabala1979 I also tried replicating in both python and bash and couldn't get the same behaviour. This behaviour has changed between V3.3.8 (Argo recognised the exit code and failed) and V3.3.9 (Argo is no longer recognising the exitCode)
Correct
The exit-code is not being used in another step, though we are expecting the step to fail when the container does not succeed. I've now updated the original post to reference an image from dockerhub that replicates this issue for further clarity. If I run the container locally we can see the expected exit code is 134 (non-zero therefore failure) Running Locally$ docker run -it --rm --entrypoint pwsh hrusby/dotnet-immediate-exception:1.0.0
Unable to find image 'hrusby/dotnet-immediate-exception:1.0.0' locally
1.0.0: Pulling from hrusby/dotnet-immediate-exception
c1ad9731b2c7: Already exists
169ba0027942: Already exists
c4c86b92f556: Already exists
e76245086e24: Already exists
0bf07af7e5b6: Already exists
85081c9db788: Already exists
5fdeb3a38fe8: Already exists
d395fb0767b4: Already exists
4aef9e22d0fd: Pull complete
Digest: sha256:e10e4d66a8c35ebd1a917f831bec5d3466292bec1861b758c573057312f087f0
Status: Downloaded newer image for hrusby/dotnet-immediate-exception:1.0.0
PowerShell 7.1.7
Copyright (c) Microsoft Corporation.
https://aka.ms/powershell
Type 'help' to get help.
PS /> echo $LastExitCode
PS /> dotnet ./Immediate-Exception.dll
Unhandled exception. System.Exception: Some Error
at Immediate_Exception.Program.Main(String[] args) in /home/harry/Development/Immediate-Exception/Program.cs:line 9
PS /> echo $LastExitCode
134
PS />Logs for the workflow running this containerapiVersion: v1
items:
- apiVersion: v1
kind: Pod
metadata:
annotations:
kubectl.kubernetes.io/default-container: main
workflows.argoproj.io/node-id: dotnet-exception-test-qmphc
workflows.argoproj.io/node-name: dotnet-exception-test-qmphc
workflows.argoproj.io/outputs: '{"artifacts":[{"name":"main-logs","s3":{"key":"dotnet-exception-test-qmphc/dotnet-exception-test-qmphc/main.log"}}]}'
creationTimestamp: "2022-08-23T17:55:20Z"
labels:
workflows.argoproj.io/completed: "true"
workflows.argoproj.io/workflow: dotnet-exception-test-qmphc
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
f:workflows.argoproj.io/outputs: {}
manager: argoexec
operation: Update
time: "2022-08-23T17:55:24Z"
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:status:
f:conditions:
k:{"type":"ContainersReady"}:
.: {}
f:lastProbeTime: {}
f:lastTransitionTime: {}
f:reason: {}
f:status: {}
f:type: {}
k:{"type":"Initialized"}:
.: {}
f:lastProbeTime: {}
f:lastTransitionTime: {}
f:reason: {}
f:status: {}
f:type: {}
k:{"type":"Ready"}:
.: {}
f:lastProbeTime: {}
f:lastTransitionTime: {}
f:reason: {}
f:status: {}
f:type: {}
f:containerStatuses: {}
f:hostIP: {}
f:initContainerStatuses: {}
f:phase: {}
f:podIP: {}
f:podIPs:
.: {}
k:{"ip":"10.244.12.79"}:
.: {}
f:ip: {}
f:startTime: {}
manager: kubelet
operation: Update
time: "2022-08-23T17:55:25Z"
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:kubectl.kubernetes.io/default-container: {}
f:workflows.argoproj.io/node-id: {}
f:workflows.argoproj.io/node-name: {}
f:labels:
.: {}
f:workflows.argoproj.io/completed: {}
f:workflows.argoproj.io/workflow: {}
f:ownerReferences:
.: {}
k:{"uid":"bec02c3a-122e-4543-a3f5-72c204e514f1"}:
.: {}
f:apiVersion: {}
f:blockOwnerDeletion: {}
f:controller: {}
f:kind: {}
f:name: {}
f:uid: {}
f:spec:
f:containers:
k:{"name":"main"}:
.: {}
f:command: {}
f:env:
.: {}
k:{"name":"ARGO_CONTAINER_NAME"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"ARGO_DEADLINE"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"ARGO_INCLUDE_SCRIPT_OUTPUT"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"ARGO_NODE_ID"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"ARGO_PROGRESS_FILE"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"ARGO_PROGRESS_FILE_TICK_DURATION"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"ARGO_PROGRESS_PATCH_TICK_DURATION"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"ARGO_TEMPLATE"}:
.: {}
f:name: {}
f:value: {}
f:image: {}
f:imagePullPolicy: {}
f:name: {}
f:resources: {}
f:terminationMessagePath: {}
f:terminationMessagePolicy: {}
f:volumeMounts:
.: {}
k:{"mountPath":"/var/run/argo"}:
.: {}
f:mountPath: {}
f:name: {}
k:{"name":"wait"}:
.: {}
f:command: {}
f:env:
.: {}
k:{"name":"ARGO_CONTAINER_NAME"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"ARGO_CONTAINER_RUNTIME_EXECUTOR"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"ARGO_DEADLINE"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"ARGO_INCLUDE_SCRIPT_OUTPUT"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"ARGO_NODE_ID"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"ARGO_POD_NAME"}:
.: {}
f:name: {}
f:valueFrom:
.: {}
f:fieldRef:
.: {}
f:apiVersion: {}
f:fieldPath: {}
k:{"name":"ARGO_POD_UID"}:
.: {}
f:name: {}
f:valueFrom:
.: {}
f:fieldRef:
.: {}
f:apiVersion: {}
f:fieldPath: {}
k:{"name":"ARGO_PROGRESS_FILE"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"ARGO_PROGRESS_FILE_TICK_DURATION"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"ARGO_PROGRESS_PATCH_TICK_DURATION"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"ARGO_TEMPLATE"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"ARGO_WORKFLOW_NAME"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"GODEBUG"}:
.: {}
f:name: {}
f:value: {}
f:image: {}
f:imagePullPolicy: {}
f:name: {}
f:resources: {}
f:terminationMessagePath: {}
f:terminationMessagePolicy: {}
f:volumeMounts:
.: {}
k:{"mountPath":"/argo/secret/argo-minio"}:
.: {}
f:mountPath: {}
f:name: {}
f:readOnly: {}
k:{"mountPath":"/var/run/argo"}:
.: {}
f:mountPath: {}
f:name: {}
f:dnsPolicy: {}
f:enableServiceLinks: {}
f:initContainers:
.: {}
k:{"name":"init"}:
.: {}
f:command: {}
f:env:
.: {}
k:{"name":"ARGO_CONTAINER_NAME"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"ARGO_CONTAINER_RUNTIME_EXECUTOR"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"ARGO_DEADLINE"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"ARGO_INCLUDE_SCRIPT_OUTPUT"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"ARGO_NODE_ID"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"ARGO_POD_NAME"}:
.: {}
f:name: {}
f:valueFrom:
.: {}
f:fieldRef:
.: {}
f:apiVersion: {}
f:fieldPath: {}
k:{"name":"ARGO_POD_UID"}:
.: {}
f:name: {}
f:valueFrom:
.: {}
f:fieldRef:
.: {}
f:apiVersion: {}
f:fieldPath: {}
k:{"name":"ARGO_PROGRESS_FILE"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"ARGO_PROGRESS_FILE_TICK_DURATION"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"ARGO_PROGRESS_PATCH_TICK_DURATION"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"ARGO_TEMPLATE"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"ARGO_WORKFLOW_NAME"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"GODEBUG"}:
.: {}
f:name: {}
f:value: {}
f:image: {}
f:imagePullPolicy: {}
f:name: {}
f:resources: {}
f:terminationMessagePath: {}
f:terminationMessagePolicy: {}
f:volumeMounts:
.: {}
k:{"mountPath":"/argo/secret/argo-minio"}:
.: {}
f:mountPath: {}
f:name: {}
f:readOnly: {}
k:{"mountPath":"/var/run/argo"}:
.: {}
f:mountPath: {}
f:name: {}
f:restartPolicy: {}
f:schedulerName: {}
f:securityContext: {}
f:terminationGracePeriodSeconds: {}
f:volumes:
.: {}
k:{"name":"argo-minio"}:
.: {}
f:name: {}
f:secret:
.: {}
f:defaultMode: {}
f:items: {}
f:secretName: {}
k:{"name":"var-run-argo"}:
.: {}
f:emptyDir: {}
f:name: {}
manager: workflow-controller
operation: Update
time: "2022-08-23T17:55:30Z"
name: dotnet-exception-test-qmphc
namespace: argo
ownerReferences:
- apiVersion: argoproj.io/v1alpha1
blockOwnerDeletion: true
controller: true
kind: Workflow
name: dotnet-exception-test-qmphc
uid: bec02c3a-122e-4543-a3f5-72c204e514f1
resourceVersion: "363918596"
selfLink: /api/v1/namespaces/argo/pods/dotnet-exception-test-qmphc
uid: 18450f73-99a0-46e8-a6f7-5e504ba45bcd
spec:
containers:
- command:
- argoexec
- wait
- --loglevel
- info
env:
- name: ARGO_POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: ARGO_POD_UID
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.uid
- name: ARGO_CONTAINER_RUNTIME_EXECUTOR
value: emissary
- name: GODEBUG
value: x509ignoreCN=0
- name: ARGO_WORKFLOW_NAME
value: dotnet-exception-test-qmphc
- name: ARGO_CONTAINER_NAME
value: wait
- name: ARGO_TEMPLATE
value: '{"name":"dotnet-exception-test","inputs":{},"outputs":{},"metadata":{},"container":{"name":"","image":"hrusby/dotnet-immediate-exception:1.0.0","command":["dotnet","Immediate-Exception.dll"],"resources":{}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"argo-minio.argo:9000","bucket":"argo-artifacts","insecure":true,"accessKeySecret":{"name":"argo-minio","key":"accesskey"},"secretKeySecret":{"name":"argo-minio","key":"secretkey"},"key":"dotnet-exception-test-qmphc/dotnet-exception-test-qmphc"}}}'
- name: ARGO_NODE_ID
value: dotnet-exception-test-qmphc
- name: ARGO_INCLUDE_SCRIPT_OUTPUT
value: "false"
- name: ARGO_DEADLINE
value: "0001-01-01T00:00:00Z"
- name: ARGO_PROGRESS_FILE
value: /var/run/argo/progress
- name: ARGO_PROGRESS_PATCH_TICK_DURATION
value: 1m0s
- name: ARGO_PROGRESS_FILE_TICK_DURATION
value: 3s
image: quay.io/argoproj/argoexec:v3.3.9
imagePullPolicy: IfNotPresent
name: wait
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /argo/secret/argo-minio
name: argo-minio
readOnly: true
- mountPath: /var/run/argo
name: var-run-argo
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: default-token-4797k
readOnly: true
- command:
- /var/run/argo/argoexec
- emissary
- --
- dotnet
- Immediate-Exception.dll
env:
- name: ARGO_CONTAINER_NAME
value: main
- name: ARGO_TEMPLATE
value: '{"name":"dotnet-exception-test","inputs":{},"outputs":{},"metadata":{},"container":{"name":"","image":"hrusby/dotnet-immediate-exception:1.0.0","command":["dotnet","Immediate-Exception.dll"],"resources":{}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"argo-minio.argo:9000","bucket":"argo-artifacts","insecure":true,"accessKeySecret":{"name":"argo-minio","key":"accesskey"},"secretKeySecret":{"name":"argo-minio","key":"secretkey"},"key":"dotnet-exception-test-qmphc/dotnet-exception-test-qmphc"}}}'
- name: ARGO_NODE_ID
value: dotnet-exception-test-qmphc
- name: ARGO_INCLUDE_SCRIPT_OUTPUT
value: "false"
- name: ARGO_DEADLINE
value: "0001-01-01T00:00:00Z"
- name: ARGO_PROGRESS_FILE
value: /var/run/argo/progress
- name: ARGO_PROGRESS_PATCH_TICK_DURATION
value: 1m0s
- name: ARGO_PROGRESS_FILE_TICK_DURATION
value: 3s
image: hrusby/dotnet-immediate-exception:1.0.0
imagePullPolicy: IfNotPresent
name: main
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/run/argo
name: var-run-argo
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: default-token-4797k
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
initContainers:
- command:
- argoexec
- init
- --loglevel
- info
env:
- name: ARGO_POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: ARGO_POD_UID
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.uid
- name: ARGO_CONTAINER_RUNTIME_EXECUTOR
value: emissary
- name: GODEBUG
value: x509ignoreCN=0
- name: ARGO_WORKFLOW_NAME
value: dotnet-exception-test-qmphc
- name: ARGO_CONTAINER_NAME
value: init
- name: ARGO_TEMPLATE
value: '{"name":"dotnet-exception-test","inputs":{},"outputs":{},"metadata":{},"container":{"name":"","image":"hrusby/dotnet-immediate-exception:1.0.0","command":["dotnet","Immediate-Exception.dll"],"resources":{}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"argo-minio.argo:9000","bucket":"argo-artifacts","insecure":true,"accessKeySecret":{"name":"argo-minio","key":"accesskey"},"secretKeySecret":{"name":"argo-minio","key":"secretkey"},"key":"dotnet-exception-test-qmphc/dotnet-exception-test-qmphc"}}}'
- name: ARGO_NODE_ID
value: dotnet-exception-test-qmphc
- name: ARGO_INCLUDE_SCRIPT_OUTPUT
value: "false"
- name: ARGO_DEADLINE
value: "0001-01-01T00:00:00Z"
- name: ARGO_PROGRESS_FILE
value: /var/run/argo/progress
- name: ARGO_PROGRESS_PATCH_TICK_DURATION
value: 1m0s
- name: ARGO_PROGRESS_FILE_TICK_DURATION
value: 3s
image: quay.io/argoproj/argoexec:v3.3.9
imagePullPolicy: IfNotPresent
name: init
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /argo/secret/argo-minio
name: argo-minio
readOnly: true
- mountPath: /var/run/argo
name: var-run-argo
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: default-token-4797k
readOnly: true
nodeName: lon2kubuat13
preemptionPolicy: PreemptLowerPriority
priority: 0
restartPolicy: Never
schedulerName: default-scheduler
securityContext: {}
serviceAccount: default
serviceAccountName: default
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- emptyDir: {}
name: var-run-argo
- name: argo-minio
secret:
defaultMode: 420
items:
- key: accesskey
path: accesskey
- key: secretkey
path: secretkey
secretName: argo-minio
- name: default-token-4797k
secret:
defaultMode: 420
secretName: default-token-4797k
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2022-08-23T17:55:22Z"
reason: PodCompleted
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2022-08-23T17:55:25Z"
reason: PodCompleted
status: "False"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2022-08-23T17:55:25Z"
reason: PodCompleted
status: "False"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2022-08-23T17:55:20Z"
status: "True"
type: PodScheduled
containerStatuses:
- containerID: docker://a6871819a36188f80b1346a5bf15d07f4b3a64fd28beb4c82fed77580d100166
image: hrusby/dotnet-immediate-exception:1.0.0
imageID: docker-pullable://hrusby/dotnet-immediate-exception@sha256:e10e4d66a8c35ebd1a917f831bec5d3466292bec1861b758c573057312f087f0
lastState: {}
name: main
ready: false
restartCount: 0
started: false
state:
terminated:
containerID: docker://a6871819a36188f80b1346a5bf15d07f4b3a64fd28beb4c82fed77580d100166
exitCode: 0
finishedAt: "2022-08-23T17:55:24Z"
reason: Completed
startedAt: "2022-08-23T17:55:23Z"
- containerID: docker://e83057277e41fd8097445d02a00daa0fbc2df787a6107a7f682709b50154d244
image: quay.io/argoproj/argoexec:v3.3.9
imageID: docker-pullable://quay.io/argoproj/argoexec@sha256:e633c7097757d9947ff79d3051eeaeb760e7e5d66b5d4d314078117d6cb18ebb
lastState: {}
name: wait
ready: false
restartCount: 0
started: false
state:
terminated:
containerID: docker://e83057277e41fd8097445d02a00daa0fbc2df787a6107a7f682709b50154d244
exitCode: 0
finishedAt: "2022-08-23T17:55:24Z"
reason: Completed
startedAt: "2022-08-23T17:55:22Z"
hostIP: 10.50.100.210
initContainerStatuses:
- containerID: docker://548d6f162c4344293b9a507540f354c27b7270fa98ff1f2ac3e02f5b5f8c2207
image: quay.io/argoproj/argoexec:v3.3.9
imageID: docker-pullable://quay.io/argoproj/argoexec@sha256:e633c7097757d9947ff79d3051eeaeb760e7e5d66b5d4d314078117d6cb18ebb
lastState: {}
name: init
ready: true
restartCount: 0
state:
terminated:
containerID: docker://548d6f162c4344293b9a507540f354c27b7270fa98ff1f2ac3e02f5b5f8c2207
exitCode: 0
finishedAt: "2022-08-23T17:55:22Z"
reason: Completed
startedAt: "2022-08-23T17:55:21Z"
phase: Succeeded
podIP: 10.244.12.79
podIPs:
- ip: 10.244.12.79
qosClass: BestEffort
startTime: "2022-08-23T17:55:20Z"
kind: List
metadata:
resourceVersion: ""
selfLink: "" |
|
Looking at the PRs for V3.3.9 the suspicion at the moment is that this problem was introduced in this commit. Specifically in main.go I can see that this is likely where the exitCode 137 was coming from in previous releases. So perhaps the errors.Exited object doesn't cover all bases. |
|
I've got the same problem. |
|
@HRusby @linke-autra looks like windows container specific. can you share the container? |
|
@sarabala1979 The container is hrusby/dotnet-immediate-exception:1.0.0 (the original post was edited to include this last night). I've run the pod locally via "State": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 139,
"Error": "",
"StartedAt": "2022-08-24T16:30:41.9617513Z",
"FinishedAt": "2022-08-24T16:30:42.0822569Z"
}Result of docker inspect[
{
"Id": "eeaa2fc34ee8c39e38ea5d50410747d7f4353fd7e8742d8787a72d2905b5bb1b",
"Created": "2022-08-24T16:30:40.9132775Z",
"Path": "dotnet",
"Args": [
"Immediate-Exception.dll"
],
"State": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 139,
"Error": "",
"StartedAt": "2022-08-24T16:30:41.9617513Z",
"FinishedAt": "2022-08-24T16:30:42.0822569Z"
},
"Image": "sha256:db1687b7e5152adbbec8f634b6d267ea69d44ee0cfd7e323d3e9013a1bb2391a",
"ResolvConfPath": "/var/lib/docker/containers/eeaa2fc34ee8c39e38ea5d50410747d7f4353fd7e8742d8787a72d2905b5bb1b/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/eeaa2fc34ee8c39e38ea5d50410747d7f4353fd7e8742d8787a72d2905b5bb1b/hostname",
"HostsPath": "/var/lib/docker/containers/eeaa2fc34ee8c39e38ea5d50410747d7f4353fd7e8742d8787a72d2905b5bb1b/hosts",
"LogPath": "/var/lib/docker/containers/eeaa2fc34ee8c39e38ea5d50410747d7f4353fd7e8742d8787a72d2905b5bb1b/eeaa2fc34ee8c39e38ea5d50410747d7f4353fd7e8742d8787a72d2905b5bb1b-json.log",
"Name": "/funny_mestorf",
"RestartCount": 0,
"Driver": "overlay2",
"Platform": "linux",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "",
"ExecIDs": null,
"HostConfig": {
"Binds": null,
"ContainerIDFile": "",
"LogConfig": {
"Type": "json-file",
"Config": {}
},
"NetworkMode": "default",
"PortBindings": {},
"RestartPolicy": {
"Name": "no",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"CapAdd": null,
"CapDrop": null,
"CgroupnsMode": "host",
"Dns": [],
"DnsOptions": [],
"DnsSearch": [],
"ExtraHosts": null,
"GroupAdd": null,
"IpcMode": "private",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "",
"Privileged": false,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": null,
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"Runtime": "runc",
"ConsoleSize": [
50,
188
],
"Isolation": "",
"CpuShares": 0,
"Memory": 0,
"NanoCpus": 0,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": [],
"BlkioDeviceReadBps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": [],
"DeviceCgroupRules": null,
"DeviceRequests": null,
"KernelMemory": 0,
"KernelMemoryTCP": 0,
"MemoryReservation": 0,
"MemorySwap": 0,
"MemorySwappiness": null,
"OomKillDisable": false,
"PidsLimit": null,
"Ulimits": null,
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0,
"MaskedPaths": [
"/proc/asound",
"/proc/acpi",
"/proc/kcore",
"/proc/keys",
"/proc/latency_stats",
"/proc/timer_list",
"/proc/timer_stats",
"/proc/sched_debug",
"/proc/scsi",
"/sys/firmware"
],
"ReadonlyPaths": [
"/proc/bus",
"/proc/fs",
"/proc/irq",
"/proc/sys",
"/proc/sysrq-trigger"
]
},
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/980b51d834ba2b2963b16d6f1c6db00bda2549afd92d7203f9531e5296ef3659-init/diff:/var/lib/docker/overlay2/51976d08019e222faaa4c0e9e6e5c3a843040fc322daacc69f6459df38dc23dd/diff:/var/lib/docker/overlay2/45f3a95f11664308abbc15d9d7bafa7eabede38ab1854b468d7799f45cf29882/diff:/var/lib/docker/overlay2/fe208a7f426f26bd6df5a8b52232cdea0056808968a1a3e15d494975d3c71bf1/diff:/var/lib/docker/overlay2/ffcffb125ff5c4d113036768d76d32e1b872c9e9b7245b6de66f9bf0366f9d65/diff:/var/lib/docker/overlay2/2bdaa7bf3c654b094c4e935fffdc290b45bbc7035234d86cb5ef4854f964bd80/diff:/var/lib/docker/overlay2/b749e0ce6e4a4d7f1b1e1d3dd8a7666cf89e4a73db92c18989d856aa717294c9/diff:/var/lib/docker/overlay2/f48ec266ee58a914a9d93142d4bb1b7550a4226b3e7c2cadf5f9061b9f6fb177/diff:/var/lib/docker/overlay2/7478859cb5f1bec1cb56f2ea0cd3bb6325563ee9a85c77fca79fbc6fa011fe68/diff:/var/lib/docker/overlay2/547555462d1118b6a81b8b7d7d51ecbd961517f6b8919c2670e18e1a81c89c44/diff",
"MergedDir": "/var/lib/docker/overlay2/980b51d834ba2b2963b16d6f1c6db00bda2549afd92d7203f9531e5296ef3659/merged",
"UpperDir": "/var/lib/docker/overlay2/980b51d834ba2b2963b16d6f1c6db00bda2549afd92d7203f9531e5296ef3659/diff",
"WorkDir": "/var/lib/docker/overlay2/980b51d834ba2b2963b16d6f1c6db00bda2549afd92d7203f9531e5296ef3659/work"
},
"Name": "overlay2"
},
"Mounts": [],
"Config": {
"Hostname": "eeaa2fc34ee8",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": true,
"AttachStderr": true,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"ASPNETCORE_URLS=",
"DOTNET_RUNNING_IN_CONTAINER=true",
"DOTNET_VERSION=5.0.17",
"ASPNET_VERSION=5.0.17",
"DOTNET_GENERATE_ASPNET_CERTIFICATE=false",
"DOTNET_SDK_VERSION=5.0.408",
"DOTNET_USE_POLLING_FILE_WATCHER=true",
"NUGET_XMLDOC_MODE=skip",
"POWERSHELL_DISTRIBUTION_CHANNEL=PSDocker-DotnetSDK-Debian-10"
],
"Cmd": [
"dotnet",
"Immediate-Exception.dll"
],
"Image": "hrusby/dotnet-immediate-exception:1.0.0",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": null,
"OnBuild": null,
"Labels": {}
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "0626aaff602d0edf8fa425e1d8d9a3455aeec95acaada7acccc861e47e5c9594",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {},
"SandboxKey": "/var/run/docker/netns/0626aaff602d",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": {
"bridge": {
"IPAMConfig": null,
"Links": null,
"Aliases": null,
"NetworkID": "5c5143eac43aede7d0659b3c79c846f0a9217f6ad03e10bf71b27ae8293dddbf",
"EndpointID": "",
"Gateway": "",
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "",
"DriverOpts": null
}
}
}
}
] |
|
And doing the same thing in kubernetes via ResultsapiVersion: v1
kind: Pod
metadata:
creationTimestamp: "2022-08-24T16:40:22Z"
labels:
run: immediate-exception-test
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:labels:
.: {}
f:run: {}
f:spec:
f:containers:
k:{"name":"immediate-exception-test"}:
.: {}
f:image: {}
f:imagePullPolicy: {}
f:name: {}
f:resources: {}
f:terminationMessagePath: {}
f:terminationMessagePolicy: {}
f:dnsPolicy: {}
f:enableServiceLinks: {}
f:restartPolicy: {}
f:schedulerName: {}
f:securityContext: {}
f:terminationGracePeriodSeconds: {}
manager: kubectl.exe
operation: Update
time: "2022-08-24T16:40:22Z"
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:status:
f:conditions:
k:{"type":"ContainersReady"}:
.: {}
f:lastProbeTime: {}
f:lastTransitionTime: {}
f:message: {}
f:reason: {}
f:status: {}
f:type: {}
k:{"type":"Initialized"}:
.: {}
f:lastProbeTime: {}
f:lastTransitionTime: {}
f:status: {}
f:type: {}
k:{"type":"Ready"}:
.: {}
f:lastProbeTime: {}
f:lastTransitionTime: {}
f:message: {}
f:reason: {}
f:status: {}
f:type: {}
f:containerStatuses: {}
f:hostIP: {}
f:phase: {}
f:podIP: {}
f:podIPs:
.: {}
k:{"ip":"10.244.8.214"}:
.: {}
f:ip: {}
f:startTime: {}
manager: kubelet
operation: Update
time: "2022-08-24T16:40:24Z"
name: immediate-exception-test
namespace: default
resourceVersion: "364366883"
selfLink: /api/v1/namespaces/default/pods/immediate-exception-test
uid: f9c28cd9-b8b9-4ab0-8ff9-36df6c8b6727
spec:
containers:
- image: hrusby/dotnet-immediate-exception:1.0.0
imagePullPolicy: IfNotPresent
name: immediate-exception-test
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: default-token-swt7z
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
nodeName: <nodeName>
preemptionPolicy: PreemptLowerPriority
priority: 0
restartPolicy: Never
schedulerName: default-scheduler
securityContext: {}
serviceAccount: default
serviceAccountName: default
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- name: default-token-swt7z
secret:
defaultMode: 420
secretName: default-token-swt7z
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2022-08-24T16:40:22Z"
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2022-08-24T16:40:22Z"
message: 'containers with unready status: [immediate-exception-test]'
reason: ContainersNotReady
status: "False"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2022-08-24T16:40:22Z"
message: 'containers with unready status: [immediate-exception-test]'
reason: ContainersNotReady
status: "False"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2022-08-24T16:40:22Z"
status: "True"
type: PodScheduled
containerStatuses:
- containerID: docker://466aa014504e1048cf121b1e30d44307746cd56324a7be7cbfae135c769ddcf5
image: hrusby/dotnet-immediate-exception:1.0.0
imageID: docker-pullable://hrusby/dotnet-immediate-exception@sha256:e10e4d66a8c35ebd1a917f831bec5d3466292bec1861b758c573057312f087f0
lastState: {}
name: immediate-exception-test
ready: false
restartCount: 0
started: false
state:
terminated:
containerID: docker://466aa014504e1048cf121b1e30d44307746cd56324a7be7cbfae135c769ddcf5
exitCode: 139
finishedAt: "2022-08-24T16:40:23Z"
reason: Error
startedAt: "2022-08-24T16:40:23Z"
hostIP: 10.50.100.118
phase: Failed
podIP: 10.244.8.214
podIPs:
- ip: 10.244.8.214
qosClass: BestEffort
startTime: "2022-08-24T16:40:22Z" |
|
@HRusby Thanks for your response. this will eliminate a few doubts. I will try to reproduce with your image |
|
@HRusby Hello, I was able to reproduce your issue by using your image and workflow. I am seeing exit code 0 on pod and workflow. @terrytangyuan Could you take a look at this issue as well and see if you can reproduce this? |
alexec
added a commit
that referenced
this issue
Sep 6, 2022
juchaosong
pushed a commit
to juchaosong/argo-workflows
that referenced
this issue
Nov 3, 2022
…rgoproj#9523) Signed-off-by: Alex Collins <alex_collins@intuit.com> Signed-off-by: juchao <juchao@coscene.io>
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Checklist
Summary
What happened/what you expected to happen?
Dotnet console app running in Argo, Exception occurs and container terminates. Argo recognises this as Success in version 3.3.9. In previous versions v3.3.7 and v3.3.8 Argo correctly recognises this as a failure. Note that if I run the container locally I receive a non-zero exit code typically, 137 or 139.
What version are you running?
V3.3.9. Tested also in V3.3.7 and V3.3.8 in which the behaviour is as expected.
Diagnostics
V3.3.9
Paste the smallest workflow that reproduces the bug. We must be able to run the workflow.
For reference, the only code being executed in the image is:
Logs from the workflow controller:
Logs from in your workflow's wait container, something like:
Message from the maintainers:
Impacted by this regression? Give it a 👍. We prioritise the issues with the most 👍.
The text was updated successfully, but these errors were encountered: