rkt is unable to fetch etcd:v3.4.4 from quay.io due to missing signature

[elemental-lf]

Bug

The problem lies with an external dependency of Typhoon, so this is more of a heads-up and a way to get input on how to proceed. If the problem persists and is reproducible by others #655 might need reverting till this is fixed.

Environment

• Platform: bare-metal
• OS: CL 2303.4.0, rkt version is 1.30.0 as shipped with CL
• Release: own fork of Typhoon

Problem

Fetching quay.io/coreos/etcd:v3.4.4 with rkt fails. I've tested this on three different hosts in two different environments with two different Internet connections. Fetching the previous version v3.4.3 works on all three. This only happens when fetching the image with rkt (i.e. in ACI format). Fetching the image with docker pull for example works as expected. Using --insecure-options=all-fetch seems to disable fetching the signature but in the end it also fails but with a different error message.

Any idea if this is a problem with Quay or with the image itself? In the former case I could try to contact support@quay.io and in the latter case I could open an issue with the etcd folks. The etcd release scripts only do a docker push so I'd assume that Quay does the ACI conversion.

Desired Behavior

Fetching the image should succeed as with previous image versions of etcd.

Steps to Reproduce

# rkt --debug fetch quay.io/coreos/etcd:v3.4.4
image: searching for app image quay.io/coreos/etcd
image: remote fetching from URL "https://quay.io/c1/aci/quay.io/coreos/etcd/v3.4.4/aci/linux/amd64/"
image: fetching image from https://quay.io/c1/aci/quay.io/coreos/etcd/v3.4.4/aci/linux/amd64/
image: keys already exist for prefix "quay.io/coreos/etcd", not fetching again
image: downloading signature from https://quay.io/c1/aci/quay.io/coreos/etcd/v3.4.4/aci.asc/linux/amd64/
image: server requested deferring the signature download
Downloading ACI: 10.5 MB of an unknown total size
image: downloading signature from https://quay.io/c1/aci/quay.io/coreos/etcd/v3.4.4/aci.asc/linux/amd64/
image: server requested deferring the signature download
fetch: 
  └─error downloading the signature file: server asked to defer the download again

 rkt --debug fetch --insecure-options=all-fetch  quay.io/coreos/etcd:v3.4.4
image: searching for app image quay.io/coreos/etcd
image: remote fetching from URL "https://quay.io/c1/aci/quay.io/coreos/etcd/v3.4.4/aci/linux/amd64/"
image: fetching image from https://quay.io/c1/aci/quay.io/coreos/etcd/v3.4.4/aci/linux/amd64/
Downloading ACI: 10.5 MB of an unknown total size
fetch: 
  └─error copying image
    └─unexpected EOF

[dghubble]

Quay has generated automatic ACI signatures for rkt --trust-keys-from-https fetches for many years. I'd suspect the feature is having issues or perhaps no longer being offered for new images. I don't believe etcd folks would be able to affect this and they've not changed anything notable about their release process.

Most likely, the direction is to shift to the docker:// transport, like Kubelet and bootstrap have already done recently in Container Linux modules. rkt's signature checking was awesome in its day, but OCI standardization didn't adopt it and checking image signatures has fallen out of favor. Even here, etcd author's arent really offering signed ACIs. Automatic generation and trust of keys was intended as a weaker form of the rkt feature. I'd prepare for Quay to drop this feature regardless, even if they've kindly kept it around for a long time.

For Container Linux folks using the tip of master, you can try #659

Note: #655 was tested under Fedora CoreOS, which would not surface this - the system containers are using podman.

[dghubble]

Thanks for testing tip

[elemental-lf]

Ah, I actually forgot about docker:// or I would have tested that. That's any easy fix. Great, thanks!