en/latest/walk-through/sidecars/

[giscus[bot]]

en/latest/walk-through/sidecars/

https://argo-workflows.readthedocs.io/en/latest/walk-through/sidecars/

[DominHa]

Does command always need to be passed in for sidecars? I am using a distroless nodejs image as a sidecar in a workflow and it doesn't come with a shell, can argo make the sidecar use the default entrypoint?

[agilgur5]

Does command always need to be passed in for sidecars?

Why not try it out?

I am using a distroless nodejs image as a sidecar in a workflow and it doesn't come with a shell

You don't need a shell to use command.
Also note that Argo itself uses distroless images.

[DominHa]

If I omit command for the sidecar I run into: failed to look-up entrypoint/cmd for image "<image-name>", you must either explicitly specify the command, or list the image's command in the index. If I add command back in, and just try to specify the same .js executable as in the Dockerfile (such as in the official NodeJS distroless example), then I run into Error: failed to start command: fork/exec <executable>.js. Is there a working example of how to do this properly with Argo workflows with a Dockerfile and matching Argo workflow yaml?

[agilgur5]

If I omit command for the sidecar I run into:

Ok that's expected if it can't look up the Docker ENTRYPOINT -- in this case I believe it's in the base image.

If I add command back in, and just try to specify the same .js executable as in the Dockerfile (such as in the official NodeJS distroless example)

Perhaps a common point of confusion here with containers -- Docker's ENTRYPOINT is equivalent to k8s's command (and Docker CMD to k8s args). See also this SO answer on the confusing terminology.

The entrypoint of that image is node, so your k8s command should be node too.

then I run into Error: failed to start command: fork/exec <executable>.js

Without node in front this error makes sense too.

[ktns]

Is there any way to make a side container in KEP753 style?

[ktns]

Wait a minute, I thought you couldn't specify initContainers directly since you were asking... but you can as you wrote above and as the Fields Reference shows. Did it not work directly setting it?

We can specify initContainers but cannot specify restartPolicy: Always to containers (not only init containers) if I understand correctly. It seems to me argo controller is overwriting restartPolicy as Never after applying the podSpecPatch...
#10973 might be relevant?

Argo has its own initContainer (called init) for the executor, so podSpecPatch might be modifying the wrong container? It would be good to see the erroneous Pod manifest that k8s rejected to determine what went wrong

How can I check manifest generated by argo?

[agilgur5]

#10973 might be relevant?

Interesting, that wasn't filed as a bug with proper logs etc, but actually says the Pod restarted when it wasn't despite the hard-coded restartPolicy: Never. So vaguely related but different issue.

I checked the code and it is still hard-coded today. And that might be necessary too in order to ensure containers get terminated by the Controller.

But it's at the Pod-level, not the container-level. I don't see that it ever applies a restartPolicy at the container-level, and initContainers and podSpecPatch are added later.

You are on an older version, but the code is largely the same here.

We can specify initContainers but cannot specify restartPolicy: Always to containers (not only init containers) if I understand correctly

As far as I understand, in k8s itself, container-level restartPolicy is currently unique to KEP753 sidecars, as was mentioned in the proposal.
Are you on a new enough k8s version? I assume the error would be different if you're not, but I'm not sure.

Argo is compiled with an older version of k8s and you're on an older version of Argo, but that error isn't coming from Argo and there is no validation error on your input. Though I wonder if that might impact the spec; omitting instead of erroring would be strange though 🤔
Argo's code predates this k8s feature, so I'm also not sure if the Controller would properly handle it if it did work. The main purpose of the feature is for Jobs, for which Workflows are a more powerful variant as-is. It's meant to be unobtrusive though, so it might work.

Did you try setting restartPolicy: Always on the initContainer spec directly? As in, without podSpecPatch?

How can I check manifest generated by argo?

The Pod never got created, right? As in, no Pod in error state in your NS that you can read?
k8s api-server logs is the first I think of off the top of my head, but you may not have access to that 😅 And network introspection would work too.

Otherwise I'm not sure if Argo has a debug option to show this, I'd have to check. The k8s client-go --gloglevel flag is exposed by the Controller which might be able to show it at a debug level.

[ktns]

Are you on a new enough k8s version? I assume the error would be different if you're not, but I'm not sure.

Yes, our cluster is recently updated to k8s 1.29 and KEP753 sidecar is running correctly when I create pods/jobs directly.

Did you try setting restartPolicy: Always on the initContainer spec directly? As in, without podSpecPatch?

Yes, I tried specifying it directly but the error message was the same.

Pod "test-kep753-sidecar-direct" is invalid: spec.initContainers[1].readinessProbe: Forbidden: may not be set for init containers without restartPolicy=Always

The Pod never got created, right?

Right, nothing shows up.

k8s api-server logs is the first I think of off the top of my head, but you may not have access to that

I don't have permissions to read that but I'll try asking admin people in my company.

[superbrothers]

I created the steps to reproduce using Kubernetes 1.30 and Argo Workflow 3.5.6. The problem can be reproduced using this combination of versions.

$ kind version
kind v0.23.0 go1.21.10 linux/amd64
$ kind create cluster
$ kubectl version
Client Version: v1.30.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.0
$ kubectl create namespace argo
namespace/argo created
$ kubectl apply -n argo -f https://github.com/argoproj/argo-workflows/releases/download/v3.5.6/install.yaml
customresourcedefinition.apiextensions.k8s.io/clusterworkflowtemplates.argoproj.io created
customresourcedefinition.apiextensions.k8s.io/cronworkflows.argoproj.io created
customresourcedefinition.apiextensions.k8s.io/workflowartifactgctasks.argoproj.io created
customresourcedefinition.apiextensions.k8s.io/workfloweventbindings.argoproj.io created
customresourcedefinition.apiextensions.k8s.io/workflows.argoproj.io created
customresourcedefinition.apiextensions.k8s.io/workflowtaskresults.argoproj.io created
customresourcedefinition.apiextensions.k8s.io/workflowtasksets.argoproj.io created
customresourcedefinition.apiextensions.k8s.io/workflowtemplates.argoproj.io created
serviceaccount/argo created
serviceaccount/argo-server created
role.rbac.authorization.k8s.io/argo-role created
clusterrole.rbac.authorization.k8s.io/argo-aggregate-to-admin created
clusterrole.rbac.authorization.k8s.io/argo-aggregate-to-edit created
clusterrole.rbac.authorization.k8s.io/argo-aggregate-to-view created
clusterrole.rbac.authorization.k8s.io/argo-cluster-role created
clusterrole.rbac.authorization.k8s.io/argo-server-cluster-role created
rolebinding.rbac.authorization.k8s.io/argo-binding created
clusterrolebinding.rbac.authorization.k8s.io/argo-binding created
clusterrolebinding.rbac.authorization.k8s.io/argo-server-binding created
configmap/workflow-controller-configmap created
service/argo-server created
priorityclass.scheduling.k8s.io/workflow-controller created
deployment.apps/argo-server created
deployment.apps/workflow-controller created

Use podSpecPatch:

$ cat workflow.yaml
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: pod-spec-patch-
spec:
  entrypoint: whalesay
  templates:
  - name: whalesay
    podSpecPatch: |
      initContainers:
      - name: test
        restartPolicy: Always
    initContainers:
    - name: test
      image: busybox
      command: [/bin/sh]
      args: [sleep, inf]
    container:
      image: docker/whalesay:latest
      command: [cowsay]
      args: ["hello world"]

$ kubectl create -f workflow.yaml
workflow.argoproj.io/pod-spec-patch-qcrhb created

We can confirm that the restartPolicy field is missing.

$ kubectl get po pod-spec-patch-qcrhb -o yaml | yq -r '.spec.initContainers[] | select (.name == "test")'
args:
  - sleep
  - inf
command:
  - /bin/sh
env:
  - name: ARGO_CONTAINER_NAME
    value: test
  - name: ARGO_TEMPLATE
    value: '{"name":"whalesay","inputs":{},"outputs":{},"metadata":{},"container":{"name":"","image":"docker/whalesay:latest","command":["cowsay"],"args":["hello world"],"resources":{}},"initContainers":[{"name":"test","image":"busybox","command":["/bin/sh"],"args":["sleep","inf"],"resources":{}}],"podSpecPatch":"initContainers:\n- name: test\n  restartPolicy: Always\n"}'
  - name: ARGO_NODE_ID
    value: pod-spec-patch-qcrhb
  - name: ARGO_INCLUDE_SCRIPT_OUTPUT
    value: "false"
  - name: ARGO_DEADLINE
    value: "0001-01-01T00:00:00Z"
  - name: ARGO_PROGRESS_FILE
    value: /var/run/argo/progress
  - name: ARGO_PROGRESS_PATCH_TICK_DURATION
    value: 1m0s
  - name: ARGO_PROGRESS_FILE_TICK_DURATION
    value: 3s
image: busybox
imagePullPolicy: Always
name: test
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
  - mountPath: /var/run/argo
    name: var-run-argo
  - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
    name: kube-api-access-fjfd4
    readOnly: true
``

[CaporalDead]

Problem still exists in 3.5.12 and k8s 1.30. Having the same issue using a cloud-sql-proxy container as initContainer. restartPolicy is not forwarded to the workflow resulting in Container is running. It is started and not ready.

Looking at https://github.dev/argoproj/argo-workflows/blob/6da0ff825cf318f518f640b5415db4df6b26656d/workflow/controller/workflowpod.go#L677 it seems that the newExecContainer function, the type does not handle the restartPolicy property

[RyanDevlin]

Commenting here that it would be great if we could support editing the restartPolicy in init containers in order to use the new K8s sidecar feature because of this bug: #10612. Sidecars on Argo Workflows are not currently terminated in a timely manner. One workaround could be using the K8s native sidecars to allow the kubelet to enforce the termination, but based on this conversation it appears that is not currently supported.