How to create builder in k8s?

[morlay]

--driver=kubernetes

[morlay]

Here are two modes to create builder in k8s

With QEMU

$ KUBECONFIG=${KUBECONFIG} \
    docker buildx create \
        --name=builder \
        --platform=linux/amd64,linux/arm64 \
        --driver=kubernetes \
        --driver-opt=namespace=buildkit,qemu.install=true

Known Issues

QEMU only work well for executing, but for compiling, it will be obviously slow and may be throw segmentation fault.

In the mode, if still want to compile binaries in docker.
please use FROM --platform=${BUILDPLATFORM} to disable QEUM for compiling stage.

Example for golang:

FROM --platform=${BUILDPLATFORM} golang:1.6 as builder

ARG TARGETARCH
RUN GOARCH=${TARGETARCH} go build -o /bin/app-linux-${TARGETARCH} ./path/to/cmd/app

FROM scratch

ARG TARGETARCH
COPY --from=builder /bin/app-linux-${TARGETARCH} /bin/app

With native nodes

# create builder `builder` and add native x86_64 node
$ KUBECONFIG=${KUBECONFIG} \
    docker buildx create \
        --name=builder \
        --platform=linux/amd64 \
        --node=builder-amd64 \
        --driver=kubernetes \
        --driver-opt=namespace=buildkit,nodeselector="beta.kubernetes.io/arch=amd64"

# append node to same builder with native aarch64 node
$ KUBECONFIG=${KUBECONFIG} \
    docker buildx create \
        --name=builder --append \
        --platform=linux/arm64 \
        --node=builder-arm64 \
        --driver=kubernetes \
        --driver-opt=namespace=buildkit,nodeselector="beta.kubernetes.io/arch=arm64"

• KUBECONFIG could be different.
• buildx create executing on a pod of multi-arch cluster, KUBECONFIG could be unset, but make sure the pod serviceAccount could access deplopments,pods,configmaps of assigned namespace

Known Issues

In this mode, docker build for different arch on matched native host.
The build time may be longer.
Even FROM --platform=${BUILDPLATFORM} defined, all stages will build for each arch.

However, it is totally native.
Projects, witch needs to build on native system, will be happy with the mode.

Tips

Once buildx create in k8s, the created deployments will not be removed until buildx rm called.
So we could set RUN --mount=type=cache for sharing common caches for different projects.

Note:

• caches are not shared across different nodes.
• the deployment name is same as --node=NAME, that mean，the buildkit deployment could be apply by kubectl for adventure usage.

However, For nodejs user.
Don't shared npm or yarn global caches (pnpm will be better), restoring caches may be slower than reinstalling.