Porter fails auth for installation image pull with Kubernetes driver

[bdegeeter]

Describe the bug

Porter does not use service account auth for private registries when run with
kubernetes driver.

This is seen when running via the Porter Operator.

Digging into the I'm hitting this error...

porter/pkg/cnab/cnab-to-oci/registry.go

Line 77 in 9b9cadd

return cnab.BundleReference{}, span.Errorf("unable to pull bundle: %w", err)

resolver := r.createResolver(insecureRegistries)

	if span.ShouldLog(zapcore.DebugLevel) {
		msg := strings.Builder{}
		msg.WriteString("Pulling bundle ")
		msg.WriteString(ref.String())
		if opts.InsecureRegistry {
			msg.WriteString(" with --insecure-registry")
		}
		span.Debug(msg.String())
	}

	bun, reloMap, digest, err := remotes.Pull(ctx, ref.Named, resolver)
	if err != nil {
		return cnab.BundleReference{}, span.Errorf("unable to pull bundle: %w", err)
	}

Which uses

porter/pkg/cnab/cnab-to-oci/registry.go

Line 250 in 9b9cadd

return remotes.CreateResolver(dockerconfig.LoadDefaultConfigFile(r.Out), insecureRegistries...)

func (r *Registry) createResolver(insecureRegistries []string) containerdRemotes.Resolver {
	return remotes.CreateResolver(dockerconfig.LoadDefaultConfigFile(r.Out), insecureRegistries...)
}

I'm wondering if the issue is the use of dockerconfig.LoadDefaultConfigFile.
From what I can tell https://pkg.go.dev/github.com/cnabio/cnab-to-oci@v0.3.6/remotes#CreateResolver

remotes.CreateResolver expects a docker cli configfile. https://pkg.go.dev/github.com/docker/cli/cli/config/configfile

To Reproduce

Steps to reproduce the behavior:

1. Install Porter Operator

2. Create Agent Image with latest kubernetes driver

   • ghcr.io/sgettys/porter-agent: v1.0.0-beta.1

3. Setup docker auth for private registry on the
   namespace default service account

   kubectl create secret docker-registry ghcrio-auth --docker-server=https://ghcr.io --docker-username=bdegeeter --docker-password=<GITHUB_READ_TOKEN> --docker-email=b.degeeter@f5.com
   kubectl patch serviceaccount default -p '{"imagePullSecrets": [{"name": "ghcrio-auth"}]}'
   
   

4. Create porter.sh/Installation with bundle from private registry

5. PorterConfig porter.yaml from this repository or gist

   apiVersion: porter.sh/v1
   kind: PorterConfig
   metadata:
     name: default
     namespace: vulcan
   spec:
     defaultSecretsPlugin: kubernetes.secrets
     defaultStorage: in-cluster-mongodb
     storage:
     - config:
         url: mongodb://<username>:<password>@porter-operator-mongodb.vulcan.svc.cluster.local
       name: in-cluster-mongodb
       plugin: mongodb
   

6. AgentConfig

   apiVersion: porter.sh/v1
   kind: AgentConfig
   metadata:
     name: default
     labels:
       porter.sh/generator: "porter-operator-bundle"
   spec:
     porterRepository: ghcr.io/get-porter/porter-agent
     porterVersion: v1.0.0-beta.1
     serviceAccount: porter-agent
     installationServiceAccount: default
     pullPolicy: Always
   

7. Run this porter installation

   apiVersion: porter.sh/v1
   kind: Installation
   metadata:
     generateName: porter-test-me-
     name: porter-test-me-42v8c
     namespace: vulcan
   spec:
     bundle:
       repository: ghcr.io/bdegeeter/private-porter-test-me
       version: v0.6.0
     name: porter-test-me-e21c0bbf
     namespace: vulcan
     parameters:
       delay: "0"
       exitStatus: "0"
     schemaVersion: 1.0.2
   

8. Installation error

   apiVersion: porter.sh/v1
   kind: AgentConfig
   metadata:
     name: default
     labels:
       porter.sh/generator: "porter-operator-bundle"
   spec:
     porterRepository: ghcr.io/get-porter/porter-agent
     porterVersion: v1.0.0-beta.1
     serviceAccount: porter-agent
     installationServiceAccount: default
     pullPolicy: Always
   

Expected behavior

Porter should be able to install via the Porter Operator using it's
kubernetes driver when the installationService account is set and has
docker auth creds.

Porter Command and Output

installation.yaml

schemaVersion: 1.0.2
name: private-porter-test-me-0b6a6a6b-bb8b-43d6-b5ba-0a4372d8fe82
namespace: vulcan
bundle:
    repository: ghcr.io/bdegeeter/private-porter-test-me
    version: v0.6.0
credentialSets:
    - porter-test-me
parameters:
    delay: "0"
    exitStatus: "0"

porter installation apply installation.yaml

Job spec

   spec:
      containers:
      - args:
        - installation
        - apply
        - installation.yaml
        env:
        - name: PORTER_RUNTIME_DRIVER
          value: kubernetes
        - name: KUBE_NAMESPACE
          value: vulcan
        - name: IN_CLUSTER
          value: "true"
        - name: LABELS
          value: porter.sh/jobType=bundle-installer porter.sh/managed=true porter.sh/resourceGeneration=1
            porter.sh/resourceKind=AgentAction porter.sh/resourceName=private-porter-test-me-hb8v2-cg8jg
            porter.sh/retry= 
        - name: JOB_VOLUME_NAME
          value: private-porter-test-me-hb8v2-cg8jg-t6tbm
        - name: JOB_VOLUME_PATH
          value: /porter-shared
        - name: CLEANUP_JOBS
          value: "false"
        - name: SERVICE_ACCOUNT
          value: default
        - name: AFFINITY_MATCH_LABELS
          value: porter.sh/resourceKind=AgentAction porter.sh/resourceName=private-porter-test-me-hb8v2-cg8jg
            porter.sh/resourceGeneration=1 porter.sh/retry=

Note: SERVICE_ACCOUNT is set for cnab kubernetes driver

Job output

Loading configuration config.yaml into /app/.porter/config.yaml
default-storage: in-cluster-mongodb
default-secrets-plugin: kubernetes.secrets
storage:
    - config:
        url: mongodb://root:MZrs5aF7sm@porter-operator-mongodb.vulcan.svc.cluster.local
      name: in-cluster-mongodb
      plugin: mongodb


porter version
porter v1.0.0-beta.1 (eaf8d0d4)
porter installation apply installation.yaml
Creating a new installation    {"installation": "vulcan/private-porter-test-me-0b6a6a6b-bb8b-43d6-b5ba-0a4372d8fe82"}
unable to pull bundle: failed to resolve bundle manifest "ghcr.io/bdegeeter/private-porter-test-me:v0.6.0": failed to authorize: failed to fetch anonymous token: unexpected status: 401 Unauthorized
Stream closed EOF for vulcan/private-porter-test-me-hb8v2-cg8jg-ctc9d-kmg64 (porter-agent)

Version

v1.0.0-beta.1

[carolynvs]

Linking the corresponding issue it operator (we need both issues since it affects porter, the operator and some of our deps)

getporter/operator#68

[schristoff]

Tested locally by doing the following steps.

• Created a token and authenticated into the container registry following these steps.
• Retagged porter-hello and pushed it to my local registry.
• Ran kubectl apply -n default -f test-installation.yaml
  (Namespace is the one specified in the test-installation.yaml below)

apiVersion: getporter.org/v1
kind: Installation
metadata:
  name: hello-llama
  namespace: default
spec:
  schemaVersion: 1.0.2
  name: mellama
  namespace: default
  bundle:
    repository:  ghcr.io/schristoff/porter-hello
    version: v1.0
  parameters:
    name: quickstart

It works! Documenting this so others can reproduce if needed. :)