Using kaniko to build and push image on gha-runner-scale-set in Kubernetes mode

[Rollson]

Hi, i was going through the containermode: Kubernetes configuration for the gha-runner-scale-set. And i wanted to implement building and pushing image using Kaniko to a registry (for e.g, Azure Container Registry). Can anyone help out on how to set this up. could not find something in the documentation. My initial idea was to set up the workflow as shown below. However i had issues when deploying the workflow.
Like this error, Run '/home/runner/k8s/index.js' shell: /home/runner/externals/node16/bin/node {0} env: can't execute '/__e/node16/bin/node': No such file or directory in the checkout action and also the initialize container step showing this error Run '/home/runner/k8s/index.js' cat: can't open '/etc/*release*': No such file or directory. Is this even the right way to do it? Thank you for your thoughts or suggestion

name: Build and Deploy

on:
  push:
    branches: [ "*" ]
  workflow_dispatch:

env:
  ACR_URL: "myacr.azurecr.io"
  KANIKO_CACHE_ARGS: "--cache=true --cache-copy-layers=true --cache-ttl=24h"

jobs:
  kaniko-build:
    runs-on: arc-runner-set 
    container:
      image: gcr.io/kaniko-project/executor:debug 
    permissions:
      contents: read 

    outputs:
      commit_sha_short: ${{ steps.commit_sha.outputs.sha_short }}
    steps:

      - uses: actions/checkout@v4.1.1

      - name: short commit sha
        id: commit_sha
        run: echo "sha_short=$(git log -1 --pretty=%h)" >> $GITHUB_OUTPUT

      - name: print commit sha
        run: echo ${{ steps.commit_sha.outputs.sha_short }}

      - name: Build and Push Image with kaniko
        run: |
          echo "{\"credHelpers\":{\"${{ env.ACR_URL }}\":\"acr-env\"}}" > /kaniko/.docker/config.json

          kaniko/executor --dockerfile="./app/Dockerfile" \
            --destination="${{ env.ACR_URL }}/nginx:${{ steps.commit_sha.outputs.sha_short }}" ${{ env.KANIKO_CACHE_ARGS }} \
            --push-retry 3 
        env: 
          AZURE_CLIENT_ID: ${{ secrets.ACR_SPN_CLIENT_ID }}
          AZURE_CLIENT_SECRET: ${{ secrets.ACR_SPN_PASSWORD }}
          AZURE_TENANT_ID: ${{ secrets.AZURE_TENANT_ID }}

[Link-]

What does your values.yaml file look like? Make sure to scrub sensitive data before sharing it.

[Rollson]

here is my runner-scaleset.yaml

## githubConfigUrl is the GitHub url for where you want to configure runners
## ex: https://github.com/myorg/myrepo or https://github.com/myorg
githubConfigUrl: "https://github.com/my-org/my-repo"

## githubConfigSecret is the k8s secrets to use when auth with GitHub API.
## You can choose to use GitHub App or a PAT token
githubConfigSecret:
  ### GitHub Apps Configuration
  ## NOTE: IDs MUST be strings, use quotes
  #github_app_id: ""
  #github_app_installation_id: ""
  #github_app_private_key: |

  ### GitHub PAT Configuration
  github_token: "my-pat-token"
## If you have a pre-define Kubernetes secret in the same namespace the gha-runner-scale-set is going to deploy,
## you can also reference it via `githubConfigSecret: pre-defined-secret`.
## You need to make sure your predefined secret has all the required secret data set properly.
##   For a pre-defined secret using GitHub PAT, the secret needs to be created like this:
##   > kubectl create secret generic pre-defined-secret --namespace=my_namespace --from-literal=github_token='ghp_your_pat'
##   For a pre-defined secret using GitHub App, the secret needs to be created like this:
##   > kubectl create secret generic pre-defined-secret --namespace=my_namespace --from-literal=github_app_id=123456 --from-literal=github_app_installation_id=654321 --from-literal=github_app_private_key='-----BEGIN CERTIFICATE-----*******'
# githubConfigSecret: pre-defined-secret

## proxy can be used to define proxy settings that will be used by the
## controller, the listener and the runner of this scale set.
#
# proxy:
#   http:
#     url: http://proxy.com:1234
#     credentialSecretRef: proxy-auth # a secret with `username` and `password` keys
#   https:
#     url: http://proxy.com:1234
#     credentialSecretRef: proxy-auth # a secret with `username` and `password` keys
#   noProxy:
#     - example.com
#     - example.org

## maxRunners is the max number of runners the autoscaling runner set will scale up to.
 maxRunners: 5

## minRunners is the min number of idle runners. The target number of runners created will be
## calculated as a sum of minRunners and the number of jobs assigned to the scale set.
 minRunners: 1

# runnerGroup: "default"

## name of the runner scale set to create.  Defaults to the helm release name
# runnerScaleSetName: ""

## A self-signed CA certificate for communication with the GitHub server can be
## provided using a config map key selector. If `runnerMountPath` is set, for
## each runner pod ARC will:
## - create a `github-server-tls-cert` volume containing the certificate
##   specified in `certificateFrom`
## - mount that volume on path `runnerMountPath`/{certificate name}
## - set NODE_EXTRA_CA_CERTS environment variable to that same path
## - set RUNNER_UPDATE_CA_CERTS environment variable to "1" (as of version
##   2.303.0 this will instruct the runner to reload certificates on the host)
##
## If any of the above had already been set by the user in the runner pod
## template, ARC will observe those and not overwrite them.
## Example configuration:
#
# githubServerTLS:
#   certificateFrom:
#     configMapKeyRef:
#       name: config-map-name
#       key: ca.crt
#   runnerMountPath: /usr/local/share/ca-certificates/

## Container mode is an object that provides out-of-box configuration
## for dind and kubernetes mode. Template will be modified as documented under the
## template object.
##
## If any customization is required for dind or kubernetes mode, containerMode should remain
## empty, and configuration should be applied to the template.
 containerMode:
   type: "kubernetes"  ## type can be set to dind or kubernetes
#   ## the following is required when containerMode.type=kubernetes
   kubernetesModeWorkVolumeClaim:
     accessModes: ["ReadWriteOnce"]
#     # For local testing, use https://github.com/openebs/dynamic-localpv-provisioner/blob/develop/docs/quickstart.md to provide dynamic provision volume with storageClassName: openebs-hostpath
     storageClassName: "disk" # using Azure Disk
     resources:
        requests:
          storage: 1Gi
#   kubernetesModeServiceAccount:
#     annotations:

## template is the PodSpec for each listener Pod
## For reference: https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#PodSpec
# listenerTemplate:
#   spec:
#     containers:
#     # Use this section to append additional configuration to the listener container.
#     # If you change the name of the container, the configuration will not be applied to the listener,
#     # and it will be treated as a side-car container.
#     - name: listener
#       securityContext:
#         runAsUser: 1000
#     # Use this section to add the configuration of a side-car container.
#     # Comment it out or remove it if you don't need it.
#     # Spec for this container will be applied as is without any modifications.
#     - name: side-car
#       image: example-sidecar

## template is the PodSpec for each runner Pod
## For reference: https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#PodSpec
template:
  ## template.spec will be modified if you change the container mode
  ## with containerMode.type=dind, we will populate the template.spec with following pod spec
  ## template:
  ##   spec:
  ##     initContainers:
  ##     - name: init-dind-externals
  ##       image: ghcr.io/actions/actions-runner:latest
  ##       command: ["cp", "-r", "-v", "/home/runner/externals/.", "/home/runner/tmpDir/"]
  ##       volumeMounts:
  ##         - name: dind-externals
  ##           mountPath: /home/runner/tmpDir
  ##     containers:
  ##     - name: runner
  ##       image: ghcr.io/actions/actions-runner:latest
  ##       command: ["/home/runner/run.sh"]
  ##       env:
  ##         - name: DOCKER_HOST
  ##           value: unix:///run/docker/docker.sock
  ##       volumeMounts:
  ##         - name: work
  ##           mountPath: /home/runner/_work
  ##         - name: dind-sock
  ##           mountPath: /run/docker
  ##           readOnly: true
  ##     - name: dind
  ##       image: docker:dind
  ##       args:
  ##         - dockerd
  ##         - --host=unix:///run/docker/docker.sock
  ##         - --group=$(DOCKER_GROUP_GID)
  ##       env:
  ##         - name: DOCKER_GROUP_GID
  ##           value: "123"
  ##       securityContext:
  ##         privileged: true
  ##       volumeMounts:
  ##         - name: work
  ##           mountPath: /home/runner/_work
  ##         - name: dind-sock
  ##           mountPath: /run/docker
  ##         - name: dind-externals
  ##           mountPath: /home/runner/externals
  ##     volumes:
  ##     - name: work
  ##       emptyDir: {}
  ##     - name: dind-sock
  ##       emptyDir: {}
  ##     - name: dind-externals
  ##       emptyDir: {}
  ######################################################################################################
  ## with containerMode.type=kubernetes, we will populate the template.spec with following pod spec
  ## template:
  ##   spec:
  ##     containers:
  ##     - name: runner
  ##       image: ghcr.io/actions/actions-runner:latest
  ##       command: ["/home/runner/run.sh"]
  ##       env:
  ##         - name: ACTIONS_RUNNER_CONTAINER_HOOKS
  ##           value: /home/runner/k8s/index.js
  ##         - name: ACTIONS_RUNNER_POD_NAME
  ##           valueFrom:
  ##             fieldRef:
  ##               fieldPath: metadata.name
  ##         - name: ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER
  ##           value: "true"
  ##       volumeMounts:
  ##         - name: work
  ##           mountPath: /home/runner/_work
  ##     volumes:
  ##       - name: work
  ##         ephemeral:
  ##           volumeClaimTemplate:
  ##             spec:
  ##               accessModes: [ "ReadWriteOnce" ]
  ##               storageClassName: "local-path"
  ##               resources:
  ##                 requests:
  ##                   storage: 1Gi
  spec:
    securityContext:
      fsGroup: 123
    containers:
      - name: runner
        image: ghcr.io/actions/actions-runner:latest
        command: ["/home/runner/run.sh"]
        env:
        - name: ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER
          value: "false"
    volumes:
      - name: work
        ephemeral:
          volumeClaimTemplate:
            spec:
              accessModes: [ "ReadWriteOnce" ]
              storageClassName: "disk" # using Azure Disk
              resources:
                requests:
                  storage: 1Gi

## Optional controller service account that needs to have required Role and RoleBinding
## to operate this gha-runner-scale-set installation.
## The helm chart will try to find the controller deployment and its service account at installation time.
## In case the helm chart can't find the right service account, you can explicitly pass in the following value
## to help it finish RoleBinding with the right service account.
## Note: if your controller is installed to only watch a single namespace, you have to pass these values explicitly.
# controllerServiceAccount:
#   namespace: arc-system
#   name: test-arc-gha-runner-scale-set-controller

[anitabee]

@Rollson you might already solved this, but this looks like a similar path that I've been through. Following spins up kaniko container in your k8s env:

    runs-on: arc-runner-set 
    container:
      image: gcr.io/kaniko-project/executor:debug 

I don't think you can check out code in that container as that executor image only has kaniko binary available. There is no node on container which actions/checkout@v4.1.1 is expecting. So you need to workaround around this to pass information; e.g., outputs from different jobs. To make situation even more challenging; Kaniko team does not recommend running the kaniko executor in another image, this looks like a design decision so they made it not so easy extensible on purpose.

[jonwest]

☝️ That's the answer, I feel like. By the time that Kaniko is called and running, the build context should already be within the workspace folder. An action should also really only do "one thing"—and what you've got here, at least to me, seems more like a job as there are multiple actions at play here.

[Rollson]

@anitabee, @jonwest thanks for the insight. I was able to get it to work. Posting it here for the sake of others

# build-with-kaniko.yml

name: Build with kaniko
on:
  push:
    branches: [ "*" ]
    paths:
      - "app/**"
      - ".github/workflows/build-with-kaniko.yml"
      
  workflow_dispatch:

env:
  KANIKO_CACHE_ARGS: "--cache=true --cache-copy-layers=true --cache-ttl=24h"

jobs:

  build-to-acr:
    runs-on: arc-runner-set 
    container:
      image: gcr.io/kaniko-project/executor:v1.20.0-debug # the kaniko image
    permissions:
      contents: read # read the repository

    steps:
      - name: Build and Push Image to ACR with kaniko
        run: |
          cat <<EOF > /kaniko/.docker/config.json
          { "credHelpers": { "${{ env.ACR_URL }}": "acr-env" } }
          EOF

          /kaniko/executor --dockerfile="./app/Dockerfile" \
            --context="${{ github.repositoryUrl }}#${{ github.ref }}#${{ github.sha }}"  \
            --destination="$ACR_URL/$NAMESPACE/nginx:$(echo ${GITHUB_SHA} | head  -c 7)" \
            ${{ env.KANIKO_CACHE_ARGS }} \
            --push-retry 5 
        env:  # authenticate to ACR via Azure Service Principal
          AZURE_CLIENT_ID: ${{ secrets.AZURE_CLIENT_ID }}
          AZURE_CLIENT_SECRET: ${{ secrets.AZURE_CLIENT_SECRET }}
          AZURE_TENANT_ID: ${{ secrets.AZURE_TENANT_ID }}
          GIT_USERNAME: ${{ github.actor }}
          GIT_PASSWORD: ${{ secrets.GITHUB_TOKEN }}
          ACR_URL: "myacr.azurecr.io"
          NAMESPACE: "frontend-apps"

[mazzy89]

The answer is not fully right. In this example you are missing the checkout step. It is not clear indeed how you bring into your self-hosted runner the Dockerfile.

[Rollson]

you don't not need a checkout step. Kaniko downloads the repo from the --context --dockerfile command specifies the directory of the Dockerfile in the repo. Authentication to github is done via the env variables: GIT_USERNAME , GIT_PASSWORD

[jonathan-fileread]

Hey @Rollson - I tried your build-with-kaniko.yml and I was still getting the docker daemon socket error - it occurs in the container: image step

[Rollson]

@jonathan-fileread, there are some changes you need to make to the runner-scale-set values file for you to get it to work. Take note of the containerMode section and the template section. Also note containerMode.type: "kubernetes"
You can read the full blog post here

# gha-runner-scale-set-value.yml

githubConfigUrl: "https://github.com/myorg/myrepo"
githubConfigSecret:
  github_token: "my-PAT"

## maxRunners is the max number of runners the autoscaling runner set will scale up to.
maxRunners: 5

## minRunners is the min number of idle runners. The target number of runners created will be
## calculated as a sum of minRunners and the number of jobs assigned to the scale set.
minRunners: 1

containerMode:
  type: "kubernetes"  ## type can be set to dind or kubernetes
  ## the following is required when containerMode.type=kubernetes
  kubernetesModeWorkVolumeClaim:
    accessModes: ["ReadWriteOnce"]
    # For local testing, use https://github.com/openebs/dynamic-localpv-provisioner/blob/develop/docs/quickstart.md to provide dynamic provision volume with storageClassName: openebs-hostpath
    storageClassName: "managed-csi" # for AKS
    resources:
      requests:
        storage: 2Gi

template:  
  spec:
    securityContext:
      fsGroup: 123 ## needed to resolve permission issues with mounted volume. https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors#error-access-to-the-path-homerunner_work_tool-is-denied
    containers:
      - name: runner
        image: ghcr.io/actions/actions-runner:latest
        command: ["/home/runner/run.sh"]
        env:
        - name: ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER
          value: "false"  ## To allow jobs without a job container to run, set ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER to false on your runner container. This instructs the runner to disable this check.
    volumes:
      - name: work
        ephemeral:
          volumeClaimTemplate:
            spec:
              accessModes: [ "ReadWriteOnce" ]
              storageClassName: "managed-csi" # for AKS
              resources:
                requests:
                  storage: 2Gi