Can not run e2e test on a modified cluster #1915
Comments
|
Hi @STX5, Were you able/are you able to reproduce and capture the logs from the errored container? My guess at this stage is that what you're doing is correct although there is no construct for defining a private registry for the containers used by the e2e test. There is an open pull request #1893 that addresses this. |
franknstyle
added
area/airgap
|
If the Sonobuoy e2e pod is using a private registry, it should be able to pull the images if the instructions are followed - https://sonobuoy.io/docs/v0.19.0/pullsecrets/ but the secret should be in the same ns as the e2e pod. PR #1893 addresses the issue when the containers spawned by the e2e process are using a private registry, as they will be created in different namespaces with no access to secrets. |
|
@franknstyle ，Thank you for your reply. I captured the logs from the errored container: The container image was pulled with command It was weird, because I had used Whatever, I switched back to my old Intel MacBook, and pushed another image to try again. It looked good, but the test was not progressing. I looked at the log in the e2e-job pod: I think maybe @Divya063 's explanation make a point. The e2e test creates other namespaces, which causes the ImagePullSecret not working. BTW, |
What steps did you take and what happened:
I try to run sonobuoy in my cluster with this command
and the result is like this
I retrived the result and found that the e2e test failed on pulling image "registry.k8s.io/conformance:v1.18.8".
I assume this means my cluster has trouble accessing the public registry, so I pushed this image to my private registry
then I used
sonobuoy gento manually create a yaml file(I have to do so, because my private registry requires ImagePullSecret)$ kubectl create secret docker-registry regcred --docker-server=alien-registry.alibaba-inc.com --docker-username={XXX} --docker-password={XXX}--namespace=sonobuoy --kubeconfig kubeconfig.txt secret/regcred created $ echo '{"ImagePullSecrets":"regcred"}' > secretconfig.json $ sonobuoy gen --config secretconfig.json --kubeconfig kubeconfig.txt --kube-conformance-image alien-registry.alibaba-inc.com/conformance:v1.18.8 -sonobuoy-image alien-registry.alibaba-inc.com/sonobuoy:v0.56.16 > test.yamlthen I started the test
$ sonobuoy run --wait --kubeconfig kubeconfig.txt -f test.yaml ... ... PLUGIN STATUS RESULT COUNT PROGRESS e2e failed failed 1 systemd-logs complete passed 4I tried to inspect the cluster when sonobuoy is running
A close look at the error pod
$ kubectl -n sonobuoy describe pod sonobuoy-e2e-job-0888f06407d84816 --kubeconfig kubeconfig.txt ... Containers: e2e: Container ID: docker://244457a29060c4ca0582dfde150c5cc416607003ab9854eb40dc127cdffed11f Image: alien-registry.alibaba-inc.com/conformance:v1.18.8 Image ID: docker-pullable://alien-registry.alibaba-inc.com/conformance@sha256:15cd6405e4baaeb3d13b25a296115783bba53dacad2c9e06ef530c24c5860ff4 Port: <none> Host Port: <none> Command: /run_e2e.sh State: Terminated Reason: Error Exit Code: 126 Started: Fri, 09 Jun 2023 15:13:35 +0800 Finished: Fri, 09 Jun 2023 15:13:35 +0800 Ready: False Restart Count: 0 ... Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 8m1s default-scheduler Successfully assigned sonobuoy/sonobuoy-e2e-job-0888f06407d84816 to cn-zhangjiakou.172.16.0.178 Normal Pulled 8m kubelet Container image "alien-registry.alibaba-inc.com/conformance:v1.18.8" already present on machine Normal Created 8m kubelet Created container e2e Normal Started 8m kubelet Started container e2e Normal Pulled 8m kubelet Container image "alien-registry.alibaba-inc.com/sonobuoy:v0.56.16" already present on machine Normal Created 8m kubelet Created container sonobuoy-worker Normal Started 8m kubelet Started container sonobuoy-worker Normal Killing 25s kubelet Stopping container sonobuoy-workerThe image was successfully pulled, but it seems that the container didn't correctly start.
I retrived the result, but no useful error message was logged.
What did you expect to happen:
Maybe more specific error message in the result file
Environment:
kubectl version):/etc/os-release):The text was updated successfully, but these errors were encountered: