Failed to download project zip file from the external devfile registry because of x509: certificate signed by unknown authority error

[tolusha]

Describe the bug

Failed to clone a project from the external devfile registry because of x509: certificate signed by unknown authority errord

Che version

next (development version)

Steps to reproduce

1. chectl server:deploy -p openshift --olm-channel next
2. oc create namespace external-devfile-registry
3. Deploy external devfile registry

DEVFILE_REGISTRY_URL="https://external-devfile-registry-external-devfile-registry$(oc get route -n openshift-console console -o "jsonpath={.spec.host}" | sed 's|console-openshift-console||')"
curl https://gist.githubusercontent.com/tolusha/c00afc4ce5f018854b9fbe4c6c8c9eb5/raw/b57253f5d748b91ce4111a733aef41dff474ba50/external_devfile_registry.yaml | sed 's|{{DEVFILE_REGISTRY_URL}}|'${DEVFILE_REGISTRY_URL}'|' | oc apply -f -

4. Configure Che oc patch checluster/eclipse-che --patch '{"spec": {"components": {"devfileRegistry": {"disableInternalRegistry": true, "externalDevfileRegistries": [{"url": "'${DEVFILE_REGISTRY_URL}'"}]}}}}' --type=merge -n eclipse-che

5. Start a python workspace

6. Observe error in a project-clone container

Expected behavior

Project is successfully cloned and workspace is stated

Runtime

OpenShift

Screenshots

No response

Installation method

chectl/next

Environment

Linux

Eclipse Che Logs

project clone container logs:


2023/07/25 14:18:55 Using temporary directory /projects/project-clone-3847050475
2023/07/25 14:18:55 Read DevWorkspace at /devworkspace-metadata/flattened.devworkspace.yaml
2023/07/25 14:18:55 Processing project python-hello-world
2023/07/25 14:18:55 Downloading project archive from https://<REDACTED>/resources/v2/python-hello-world.zip
2023/07/25 14:18:55 Encountered error while setting up project python-hello-world: failed to download archive: Get "https://<REDACTED>/resources/v2/python-hello-world.zip": x509: certificate signed by unknown authority

### Additional context

_No response_

[tolusha]

Probably related to the fact the downloading zip does not take care of certificates.
https://github.com/devfile/devworkspace-operator/blob/main/project-clone/internal/zip/setup.go#L86

[ibuziuk]

@amisevsk @l0rd could you please take it for the next sprint?

[l0rd]

For the record we also commented that this could be related to #22370 (if the openshift cluster has cluster wide proxy configured then the download may still fail).

[amisevsk]

I spent a bit of time looking into this: the project-clone container only uses certificates from the system pool by default, and Che mounts additional certificates to /etc/secret/che/ca.crt, which cannot be used by project clone by default (since it's a che-specific path).

We can add additional certificates to the pool, but none of the options I see are ideal:

1. We can introduce a new automount annotation for certificates, e.g. controller.devfile.io/certificate-authority, to automate mounting a secret as a certificate to a defined path.
2. We can define a known directory for additional certificates, e.g. /certificates, and read all certificates in that directory by default when using project clone.
3. We can introduce some way to configure project-clone to check additional directories for certificates

These options would all require Che-side changes as well, so I'm not sure the best way to proceed.

Our main problem here is that there's no defined way to specify additional certs for project-clone, and certificate mount paths aren't consistent enough to have a one-size-fits-all solution. Since project-clone is a go-based project, it will automatically check the following directories:

var certDirectories = []string{
	"/etc/ssl/certs",               // SLES10/SLES11, https://golang.org/issue/12139
	"/etc/pki/tls/certs",           // Fedora/RHEL
	"/system/etc/security/cacerts", // Android
}

(ref: https://go.dev/src/crypto/x509/root_linux.go)
so it is possible to workaround this issue by copying the Che-created cert to e.g. /etc/ssl/certs/:

❯ oc get secret che-server-cert -o yaml \
  | yq -Y '.metadata.name = "test-server-cert"
    | .metadata.annotations."controller.devfile.io/mount-as" = "subpath"
    | .metadata.annotations."controller.devfile.io/mount-path" = "/etc/ssl/certs/"
    | .data = {"che-ca.crt": .data."ca.crt"}' \
  | oc apply -f -

[amisevsk]

PR devfile/devworkspace-operator#1161 configures project-clone to read any certs in /public-certs, which appears to be mounted by Che in workspaces. I've tested the reproducer against my PR and it resolves the issue (though I'm not 100% sure on how reliable /public-certs is from a Che perspective.

To test the PR changes, edit the DWO CSV to change the RELATED_IMAGE_project_clone env var to quay.io/amisevsk/project-clone:self-signed

[amisevsk]

Closing this issue as completed as devfile/devworkspace-operator#1161 has been resolved. Feel free to reopen issue if something has been missed.