add --container-runtime-endpoint for node tester

[elieser1101]

Fixes #279

Pass the CONTAINER_RUNTIME_ENDPOINT so the makefile can set up properly the value.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: elieser1101
Once this PR has been reviewed and has the lgtm label, please assign xmcqueen for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

Hi @elieser1101. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[aojea]

/ok-to-test

[elieser1101]

/test pull-kubetest2-gce-node-e2e
/test pull-kubetest2-gke-up-down-multicluster

[aojea]

@elieser1101 where you able to validate manually that this works correctly?

[elieser1101]

I was able to validate locally that the value get replaced when passed and kept the default /run/containerd/containerd.sock when the flag is not set

[aojea]

I was able to validate locally that the value get replaced when passed and kept the default /run/containerd/containerd.sock when the flag is not set

/hold

#279 (comment)

It does not seem necessary to set this flag, at least I can not find the impact of not setting it, I also see some jobs passing on the history

[aojea]

If I take this job per example https://storage.googleapis.com/kubernetes-ci-logs/pr-logs/pull/128092/pull-kubernetes-node-crio-cgrpv2-imagefs-e2e-kubetest2/1860277072312668160/artifacts/n1-standard-2-fedora-coreos-41-20241027-3-0-gcp-x86-64-1037a7f5/kubelet.log

I can see how kubelet starts with the right value

I1123 11:15:37.951691 3541 flags.go:64] FLAG: --container-runtime-endpoint="unix:///var/run/crio/crio.sock"

[kannon92]

I agree with @aojea. Looking at the logs for these jobs I also see a lot of logs for cri-o and none for containerd.

[aojea]

I'm happy to spend some time helping to get this going, just find one occurrence and link it so we can investigate the problem, but it does not seem to be the selection of the endpoint, otherwise no crio test will work

[kannon92]

Digging into https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/directory/pull-kubernetes-node-crio-cgrpv2-splitfs-e2e-kubetest2/1863554529790791680.

I see that the tests that are failing seem to be hitting a selinux issue.

https://storage.googleapis.com/kubernetes-ci-logs/pr-logs/pull/120459/pull-kubernetes-node-crio-cgrpv2-splitfs-e2e-kubetest2/1863554529790791680/artifacts/n1-standard-2-fedora-coreos-41-20241109-3-0-gcp-x86-64-f11f7559/n1-standard-2-fedora-coreos-41-20241109-3-0-gcp-x86-64-f11f7559-ginkgo.log

• [FAILED] [6.205 seconds]
[sig-node] [NodeConformance] Containers Lifecycle when Running a pod with init containers and regular containers, restartPolicy=Never when A regular container has a PreStop hook when A regular container fails a startup probe [It] should call the container's preStop hook and terminate it if its startup probe fails [sig-node, NodeConformance]
k8s.io/kubernetes/test/e2e_node/container_lifecycle_test.go:658

  Timeline >>
  STEP: Creating a kubernetes client @ 12/02/24 12:19:58.333
  STEP: Building a namespace api object, basename containers-lifecycle-test @ 12/02/24 12:19:58.333
  I1202 12:19:58.341703 3269 framework.go:275] Skipping waiting for service account
  STEP: Waiting for the pod to complete @ 12/02/24 12:19:58.359
  STEP: Parsing results @ 12/02/24 12:20:04.377
  I1202 12:20:04.388371 3269 container_lifecycle_pod_construction.go:342] **Unexpected error: Failed to parse the timestamp, log: "sh: error while loading shared libraries: /lib/libc.so.6: cannot apply additional memory protection after relocation: Permission denied":** 
      <*time.ParseError | 0xc0014eb9f0>: 
      parsing time "sh:" as "2006-01-02T15:04:05Z07:00": cannot parse "sh:" as "2006"
      {
          Layout: "2006-01-02T15:04:05Z07:00",
          Value: "sh:",
          LayoutElem: "2006",
          ValueElem: "sh:",
          Message: "",
      }

[elieser1101]

Hi folks, I think you are right, this is not going to solve the issue of the linked jobs, after looking at the logs for the kubelet it seem to actually use the crio runtime-endpoint...

On the other hand, could be useful to have the CONTAINER_RUNTIME_ENDPOINT to be explicit to avoid confusion

Ginkgo ran 1 suite in 22m45.28941157s

Test Suite Failed

Failure Finished Test Suite on Host n1-standard-2-fedora-coreos-41-20241109-3-0-gcp-x86-64-f11f7559. Refer to artifacts directory for ginkgo log for this host.
command [ssh -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o CheckHostIP=no -o StrictHostKeyChecking=no -o ServerAliveInterval=30 -o LogLevel=ERROR -i /root/.ssh/google_compute_engine core@34.145.95.157 -- sudo /bin/bash -c 'cd /tmp/node-e2e-20241202T121901 && set -o pipefail; timeout -k 30s 10800.000000s ./ginkgo -timeout=24h -nodes=8  -focus="\[NodeConformance\]|\[NodeFeature:.+\]|\[NodeFeature\]"  -skip="\[Flaky\]|\[Slow\]|\[Serial\]"  --no-color -v --timeout=120m ./e2e_node.test -- --system-spec-name= --system-spec-file= --extra-envs= --runtime-config= --v 4 --node-name=n1-standard-2-fedora-coreos-41-20241109-3-0-gcp-x86-64-f11f7559 --report-dir=/tmp/node-e2e-20241202T121901/results --report-prefix=fedora --image-description="fedora-coreos-41-20241109-3-0-gcp-x86-64" --kubelet-flags="--cluster-domain=cluster.local" --dns-domain="cluster.local" --prepull-images=false  --container-runtime-endpoint=unix:///run/containerd/containerd.sock --ginkgo.timeout=3h --service-feature-gates="KubeletSeparateDiskGC=true" --feature-gates="KubeletSeparateDiskGC=true" --container-runtime-endpoint=unix:///var/run/crio/crio.sock --container-runtime-process-name=/usr/local/bin/crio --container-runtime-pid-file= --kubelet-flags="--cgroup-driver=systemd --cgroups-per-qos=true --cgroup-root=/ --runtime-cgroups=/system.slice/crio.service --kubelet-cgroups=/system.slice/kubelet.service" --extra-log="{\"name\": \"crio.log\", \"journalctl\": [\"-u\", \"crio\"]}" 2>&1 | tee -i /tmp/node-e2e-20241202T121901/results/n1-standard-2-fedora-coreos-41-20241109-3-0-gcp-x86-64-f11f7559-ginkgo.log'] failed with error: exit status 1

currently it has two times the flag --container-runtime-endpoint which is confusing

Anyhow im happy to close this issue if you consider this is not necessary