Test PR 225 (install/pre-install: Update containerd)

[wainersm]

I'm trying to merge #179 before PR #225 so that the later would be tested before merge. However, PR #179 is blocked due to what seems network issues on SEV and TDX machine when it comes to build the pre-install image.

As we are approaching the 0.7.0 release I think it is more prudent to unblock the merge of #225. So this PR, where I pilled up #225 on #179 then I will run the non-TEE tests jobs. After the merge of #225 and when a new pre-install image is built then we will have the opportunity to test the updated containerd on SEV and TDX (as the pre-install image won't be built on those machines I expect the test suite to just run).

That's my rationale for this PR. Marking as do-not-merge.

[wainersm]

/test-kata-qemu
/test-kata-clh

[wainersm]

On both kata-clh and kata-qemu jobs we are getting connectivety issues to build the pre-install image:

09:42:53 #9 96.23 E: Failed to fetch https://packages.cloud.google.com/apt/dists/kubernetes-xenial/main/binary-s390x/by-hash/SHA256/5347fddf3a45c1f4bef10ba934363533f13544e91c325ce8a276bcfbf3ef1f01  404  Not Found [IP: 172.253.115.101 443]
09:42:53 #9 96.23 E: Some index files failed to download. They have been ignored, or old ones used instead.
09:42:53 #9 ERROR: process "/bin/bash -o pipefail -c apt-get update && apt-get install -y --no-install-recommends apt-transport-https ca-certificates curl xz-utils systemd && mkdir -p /etc/apt/keyrings/ && curl -fsSLo /etc/apt/keyrings/kubernetes-archive-keyring.gpg https://dl.k8s.io/apt/doc/apt-key.gpg && echo \"deb [signed-by=/etc/apt/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main\" | tee /etc/apt/sources.list.d/kubernetes.list && apt-get update && apt-get install -y --no-install-recommends kubectl && apt-get clean && rm -rf /var/lib/apt/lists/ && curl -fOL --progress-bar [https://github.com/confidential-containers/containerd/releases/download/v${VERSION}/containerd-${VERSION}-linux-${ARCH}.tar.gz](https://github.com/confidential-containers/containerd/releases/download/v$%7BVERSION%7D/containerd-$%7BVERSION%7D-linux-$%7BARCH%7D.tar.gz) && tar xvzpf containerd-${VERSION}-linux-${ARCH}.tar.gz -C ${DESTINATION}/opt/confidential-containers" did not complete successfully: exit code: 100
09:42:53 ------

[jepio]

This might be a long shot but can you replace the repo url with:
http://packages.cloud.google.com/apt/ instead of https://apt.kubernetes.io/ (the http is intentional).

Otherwise (if that doesn't help) lets do like you suggested: fetch kubectl with curl directly.

[wainersm]

This might be a long shot but can you replace the repo url with: http://packages.cloud.google.com/apt/ instead of https://apt.kubernetes.io/ (the http is intentional).

Otherwise (if that doesn't help) lets do like you suggested: fetch kubectl with curl directly.

hmmm... trying this solution first: 91e2c8a

[wainersm]

/test-kata-qemu
/test-kata-clh

[wainersm]

Forgot to install gpg in the container... let's see:

/test-kata-qemu
/test-kata-clh

[wainersm]

Updated this PR to include the commit cf3aaef which changed the pre-install Dockerfile to pull the kubectl binary, instead of installing via package which has been a nightmare.

/test-kata-qemu
/test-kata-clh

[stevenhorsman]

/test

[stevenhorsman]

SEV tests have failed with:

10:37:06 + docker manifest push --insecure localhost:5000/container-engine-for-cc-payload:latest
10:37:06 failed to put manifest localhost:5000/container-engine-for-cc-payload:latest: errors:
10:37:06 manifest blob unknown: blob unknown to registry
10:37:06 manifest blob unknown: blob unknown to registry
10:37:06 manifest blob unknown: blob unknown to registry
10:37:06 manifest blob unknown: blob unknown to registry
10:37:06 manifest blob unknown: blob unknown to registry
10:37:06 manifest blob unknown: blob unknown to registry

and

11:31:13 fatal: [localhost]: FAILED! => {"changed": false, "msg": "Failure downloading https://go.dev/dl/go1.18.5.linux-amd64.tar.gz, Request failed: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)>"}

I've requeued, so hoping for third time lucky

[stevenhorsman]

The third sev attempt failed with:

13:07:22 + docker manifest push --insecure localhost:5000/container-engine-for-cc-payload:8056fd0d9fc0b4e2bb3201bc939b85ad414ce9a4
13:07:22 failed to put manifest localhost:5000/container-engine-for-cc-payload:8056fd0d9fc0b4e2bb3201bc939b85ad414ce9a4: errors:
13:07:22 manifest blob unknown: blob unknown to registry
13:07:22 manifest blob unknown: blob unknown to registry
13:07:22 
13:07:22 make: *** [Makefile:8: containerd-container-image] Error 1

again, so I'm not going to try and re-do it a fourth time. @UnmeshDeodhar @ryansavino - do you have any ideas? Any reason why the node can't built the new container image?

[wainersm]

The third sev attempt failed with:

13:07:22 + docker manifest push --insecure localhost:5000/container-engine-for-cc-payload:8056fd0d9fc0b4e2bb3201bc939b85ad414ce9a4
13:07:22 failed to put manifest localhost:5000/container-engine-for-cc-payload:8056fd0d9fc0b4e2bb3201bc939b85ad414ce9a4: errors:
13:07:22 manifest blob unknown: blob unknown to registry
13:07:22 manifest blob unknown: blob unknown to registry
13:07:22 
13:07:22 make: *** [Makefile:8: containerd-container-image] Error 1

again, so I'm not going to try and re-do it a fourth time. @UnmeshDeodhar @ryansavino - do you have any ideas? Any reason why the node can't built the new container image?

It is failing to push the image to a local registry. I've almost sure I saw that error before but can't remember the reason. As I commented on PR #225 I think we can disregard the sev results, merge that PR then test again with the new built image.

[wainersm]

I am going to close this as the PR #225 is merged already.