Add DockerHub credentials such that all action requests to dockerhub are authenticated

[gotchipete]

What would you like added?

A way to add our dockerhub credentials to the action runners such that we won't be rate limited by dockerhub pulls that our github actions perform.

Why is this needed?

We are facing rate limits even after using a dockerhub authentication step in our github action workflows, because we use Docker powered open source actions in our workflows. (This is because container pull for the custom actions happens at the first step of the build, even before our docker login step, and thus rate limiting still prevails.)

Additional context

In this article (search for silver bullet) - they used an ec2 post install script to authenticate to dockerhub after their ec2s start up such that all requests from the runner to dockerhub are authenticated. Is something similar possible here?

[github-actions]

Hello! Thank you for filing an issue.

The maintainers will triage your issue shortly.

In the meantime, please take a look at the troubleshooting guide for bug reports.

If this is a feature request, please review our contribution guidelines.

[lhaussknecht]

We are mounting a secret in the docker container that contains the credentials for private registries.

       - name: docker
          resources: {}
          volumeMounts:
            - mountPath: /home/runner/.docker/
              name: docker-secret
              readOnly: true


      volumes:
        - name: docker-secret
          secret:
            items:
              - key: .dockerconfigjson
                path: config.json
            secretName: regcred

The secret has to be a kubernetes.io/dockerconfigjson.

[gotchipete]

We are mounting a secret in the docker container that contains the credentials for private registries.

       - name: docker
          resources: {}
          volumeMounts:
            - mountPath: /home/runner/.docker/
              name: docker-secret
              readOnly: true


      volumes:
        - name: docker-secret
          secret:
            items:
              - key: .dockerconfigjson
                path: config.json
            secretName: regcred

The secret has to be a kubernetes.io/dockerconfigjson.

Hmm ok. Apologies if this is a dumb question, but in what file would this go? I installed the runners via helm, with my only yaml file specifying RunnerDeployments and HorizontalRunnerAutoscalers, I didn't specify any containers. Oh, or do I place the above (or something similar) into custom volume mounts, per https://github.com/actions/actions-runner-controller/blob/master/docs/using-custom-volumes.md ?

... something like this?

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: my-action-runner-deploy
  namespace: actions-runners
spec:
  replicas: 3
  template:
    spec:
      repository: myorg/foo-repo
      volumeMounts:
        - mountPath: /home/runner/.docker/
          name: docker-secret
          readOnly: true
      volumes:
        - name: docker-secret
          secret:
            items:
              - key: .dockerconfigjson
                path: config.json
            secretName: regcred

[lhaussknecht]

That looks good. To create a secret: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/ .

I think this sould be a minimal config:

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerSet
metadata:
  name: default-runner-set
spec:
  ephemeral: true
  replicas: 2
  organization: org
  group: xxx
  labels:
    - yyy
  selector:
    matchLabels:
      app: default-runner-set
  template:

    spec:
      containers:
        - name: runner
          volumeMounts:
            - name: docker-secret
              mountPath: "/home/runner/.docker/"
              readOnly: true
      volumes:
        - name: docker-secret
          secret:
            secretName: regcred
            items:
              - key: .dockerconfigjson
                path: config.json

[gotchipete]

Thanks @lhaussknecht ! Per this in your example:

    spec:
      containers:
        - name: runner

... is "runner" a reference the pre-existing/OOTB (e.g. comes with ARC) container? Or is it a container I have to create?

Sorry, still unsure if this solution presumes a DinD scenario or not (which I don't have currently).

Thank u again 🙏

[tobgen]

I did try this approach with mounting the docker credentials as a volume.
But, now I do run into this:

runner-bmv2r-v9dqm:/$ docker login
Authenticating with existing credentials...
WARNING! Your password will be stored unencrypted in /home/runner/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Error saving credentials: open /home/runner/.docker/config.json3336164637: read-only file system

This is kinda expected, since you can't write to a secret.
Do anyone have solution for this?

[roulettedares]

I did try this approach with mounting the docker credentials as a volume. But, now I do run into this:

runner-bmv2r-v9dqm:/$ docker login
Authenticating with existing credentials...
WARNING! Your password will be stored unencrypted in /home/runner/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Error saving credentials: open /home/runner/.docker/config.json3336164637: read-only file system

This is kinda expected, since you can't write to a secret. Do anyone have solution for this?

you can try creating an emptyDir and mounting it at `/home/runner/.docker in combination with this fsGroup

  spec:
    template:
      spec:
        securityContext:
          fsGroup: 1000

and mounting config.json as a subPath

          - mountPath: /home/runner/.docker/config.json
            name: container-registry
            subPath: .config.json

but, it might not work if config.json needs to be writable

i was thinking about using an init-container in my runnerset spec to download a credential helper or creating an action that does this, but i have resorted to using gke workload identity to generate an access_token and piping it to docker login for the time being

jobs:
  container-build:
    runs-on:
      - self-hosted
      - 23cpu-7g
    steps:
      - name: Set up Docker Context for Buildx
        run: docker context create arc
      - uses: docker/setup-buildx-action@f03ac48505955848960e80bbb68046aa35c7b9e7 # v2.4.1
        with:
          endpoint: arc
      - id: gcr-auth
        run: |-
          access_token="$(curl -s --retry 5 \
            "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token" \
            -H "Metadata-Flavor: Google" \
            | jq -r .access_token)"
          echo "access_token=$access_token" >> $GITHUB_OUTPUT
          echo "::add-mask::$access_token"
      - name: Login to GCR
        uses: docker/login-action@v2
        with:
          registry: gcr.io
          username: oauth2accesstoken
          password: ${{ steps.gcr-auth.outputs.access_token }}
      - name: Build and push Docker images
        uses: docker/build-push-action@3b5e8027fcad23fda98b2e3ac259d8d67585f671 #v4.0.0
        with:
          push: true
          tags: gcr.io/my-org/java:${{ env.VERSION }}

[thepoppingone]

If this helps anyone, assuming you already the regcred secret in the k8s cluster, you can use this terraform snippet to create the RunnerDeployment resource to create the specs required for mounting the dockerconfig and yet make it writable.

Many thanks to @joshuasimon-taulia and @lhaussknecht for the above tips leading to my solution

resource "kubernetes_manifest" "github_ent_runners" {

  manifest = {
    apiVersion = "actions.summerwind.dev/v1alpha1"
    kind       = "RunnerDeployment"

    metadata = {
      name      = "name-runner-deployment"
      namespace = helm_release.release.namespace
    }

    spec = {
      template = {
        spec = {
          enterprise         = enterpriseName
          serviceAccountName = serviceAccountName
          group              = group
          imagePullPolicy    = "IfNotPresent"
          securityContext = {
            fsGroup = 1000
          }
          labels      = label
          resources   = resources
          tolerations = tolerations
          affinity    = affinity
          volumeMounts = [
            {
              mountPath = "/home/runner/.docker/config.json"
              subPath   = "config.json"
              name      = "docker-secret"
            },
            {
              mountPath = "/home/runner/.docker"
              name      = "docker-config-volume"
            },
          ]
          volumes = [
            {
              name = "docker-secret"
              secret = {
                secretName = "regcred"
                items = [
                  {
                    key  = ".dockerconfigjson"
                    path = "config.json"
                  }
                ]
              }
            },
            {
              name     = "docker-config-volume"
              emptyDir = {}
            }
          ]

        }
      }
    }
  }

  depends_on = [
    helm_release.release
  ]
}

[thepoppingone]

A problem with the above solution is that, the docker config json files become read-only and causes ECR login errors.

[thepoppingone]

template = {
        spec = {
          enterprise = each.value.name
          initContainers = [
            {
              command = ["sh", "-c", "cat /home/runner/config.json > /home/runner/.docker/config.json"]
              image   = "alpine"
              securityContext = {
                fsGroup = 1000
              }
              name = "dockerconfigwriter"
              volumeMounts = [
                {
                  mountPath = "/home/runner/config.json"
                  subPath   = "config.json"
                  name      = "docker-secret"
                },
                {
                  mountPath = "/home/runner/.docker"
                  name      = "docker-config-volume"
                },
              ]
            }
          ]
          dockerdWithinRunnerContainer = true
          image                        = "summerwind/actions-runner-dind"
          serviceAccountName           = var.service_account_name
          group                        = each.value.group
          imagePullPolicy              = "IfNotPresent"
          labels                       = [each.value.label]
          resources                    = each.value.resources
          tolerations                  = each.value.tolerations
          affinity                     = each.value.affinity
          volumeMounts = [
            {
              mountPath = "/home/runner/config.json"
              subPath   = "config.json"
              name      = "docker-secret"
            },
            {
              mountPath = "/home/runner/.docker"
              name      = "docker-config-volume"
            },
          ]
          volumes = [
            {
              name = "docker-secret"
              secret = {
                secretName = "regcred"
                items = [
                  {
                    key  = ".dockerconfigjson"
                    path = "config.json"
                  }
                ]
              }
            },
            {
              name     = "docker-config-volume"
              emptyDir = {}
            },
          ]

        }
      }
    }

Managed to fix it with initContainers as suggested

[gfrid]

@thepoppingone is this working?

[gfrid]

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: runner-deployment
spec:
  template:
    spec:
      securityContext:
        fsGroup: 1000
      organization: my-org
      imagePullSecrets:
        - name: regcred
      containers:
        - name: runner
          volumeMounts:
            - name: docker-secret
              mountPath: /home/runner/config.json
              subPath: config.json
            - name: docker-config-volume
              mountPath: /home/runner/.docker
          env: []
          resources:
            limits:
              cpu: "4.0"
              memory: "8Gi"
            requests:
              cpu: "1.0"
              memory: "2Gi"
      initContainers:
        - name: dockerconfigwriter
          image: alpine
          command:
            - sh
            - -c
            - cat /home/runner/config.json > /home/runner/.docker/config.json
          volumeMounts:
            - name: docker-secret
              mountPath: /home/runner/config.json
              subPath: config.json
            - name: docker-config-volume
              mountPath: /home/runner/.docker
      volumes:
        - name: docker-secret
          secret:
            secretName: regcred
            items:
              - key: .dockerconfigjson
                path: config.json
        - name: docker-config-volume
          emptyDir: {}

this is verified as working:

https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

[moonape1226]

@gfrid
Thanks. Your solution works like a charm.

[patilvishal1683]

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: runner-deployment
spec:
  template:
    spec:
      securityContext:
        fsGroup: 1000
      organization: my-org
      imagePullSecrets:
        - name: regcred
      containers:
        - name: runner
          volumeMounts:
            - name: docker-secret
              mountPath: /home/runner/config.json
              subPath: config.json
            - name: docker-config-volume
              mountPath: /home/runner/.docker
          env: []
          resources:
            limits:
              cpu: "4.0"
              memory: "8Gi"
            requests:
              cpu: "1.0"
              memory: "2Gi"
      initContainers:
        - name: dockerconfigwriter
          image: alpine
          command:
            - sh
            - -c
            - cat /home/runner/config.json > /home/runner/.docker/config.json
          volumeMounts:
            - name: docker-secret
              mountPath: /home/runner/config.json
              subPath: config.json
            - name: docker-config-volume
              mountPath: /home/runner/.docker
      volumes:
        - name: docker-secret
          secret:
            secretName: regcred
            items:
              - key: .dockerconfigjson
                path: config.json
        - name: docker-config-volume
          emptyDir: {}

this is verified as working:

https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

Hi , How can i check the logs Login Succeeded?

[john-yacuta-submittable]

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: runner-deployment
spec:
  template:
    spec:
      securityContext:
        fsGroup: 1000
      organization: my-org
      imagePullSecrets:
        - name: regcred
      containers:
        - name: runner
          volumeMounts:
            - name: docker-secret
              mountPath: /home/runner/config.json
              subPath: config.json
            - name: docker-config-volume
              mountPath: /home/runner/.docker
          env: []
          resources:
            limits:
              cpu: "4.0"
              memory: "8Gi"
            requests:
              cpu: "1.0"
              memory: "2Gi"
      initContainers:
        - name: dockerconfigwriter
          image: alpine
          command:
            - sh
            - -c
            - cat /home/runner/config.json > /home/runner/.docker/config.json
          volumeMounts:
            - name: docker-secret
              mountPath: /home/runner/config.json
              subPath: config.json
            - name: docker-config-volume
              mountPath: /home/runner/.docker
      volumes:
        - name: docker-secret
          secret:
            secretName: regcred
            items:
              - key: .dockerconfigjson
                path: config.json
        - name: docker-config-volume
          emptyDir: {}

this is verified as working:

https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

Hi , How can i check the logs Login Succeeded?

@gfrid @moonape1226
I also would like to know and see the logs that the login succeeded. Thanks.

[john-yacuta-submittable]

@patilvishal1683 I wanted to point out that after using the code snippet from @gfrid I was able to verify in the container that I was logged in via the Docker CLI like below. Cheers!

$ kubectl exec -n <NAMESPACE> <POD_NAME> -- docker info | grep Username
Defaulted container "runner" out of: runner, docker, dockerconfigwriter (init)
 Username: <USERNAME>

[jonathan-fileread]

Hey y'all, How do you configure the config.json to apply for both ACR and dockerhub?
I've baked it in my github actions, as I'm running containermode="kubernetes" and kaniko (so not sure if volume mounts work) i know for a fact ACR works (i can push to a private repo) - but i'm not sure if docker login successfully authenticates. I'm currently getting rate limited cause i pull a lot of images

the dockerhub_creds contain base64 encoded :<dockerhub_pat>

Would love some assistance! Thank you! :)