Any way to make --cache-to output smaller?

[kohenkatz]

I was very happy to see support for --cache-from and --cache-to appear in buildah 1.27.0. (I used to use Kaniko with its --cache argument which did the same thing, but Kaniko does not have proper support for RUN --mount....)

The problem I have is with the size of the cache (and therefore also the build time).

Here's an example file:

# syntax = docker/dockerfile:1.2

# get modules, if they don't change the cache can be used for faster builds
FROM docker.io/library/golang:1.18 AS base
ENV GO111MODULE=on
ENV CGO_ENABLED=0
ENV GOOS=linux
ENV GOARCH=amd64
WORKDIR /src
COPY go.* .
RUN --mount=type=cache,target=/go/pkg/mod \
	--mount=type=secret,id=netrc,target=/root/.netrc \
    go mod download

# build the application
FROM base AS build
WORKDIR /src
# temp mount all files instead of loading into image with COPY
# temp mount module cache
# temp mount go build cache
RUN --mount=type=bind,target=/src \
    --mount=type=cache,target=/go/pkg/mod \
    --mount=type=cache,target=/root/.cache/go-build \
    --mount=type=secret,id=netrc,target=/root/.netrc \
    go build -ldflags="-w -s" -o /app/liveserver ./cmd/liveserver/*.go

# Import the binary from build stage
FROM gcr.io/distroless/static:nonroot as liveserver
COPY --from=build /app/liveserver /
# this is the numeric version of user nonroot:nonroot to check runAsNonRoot in kubernetes
USER 65532:65532
ENTRYPOINT ["/liveserver"]

Here's the command I use to run it:

buildah bud --secret id=netrc,src=/root/.netrc --cache-from registry.example.com/group/project/cache --cache-to registry.example.com/group/project/cache --layers -f Dockerfile -t liveserver --target liveserver .

I first noticed the issue because the build was running extremely slowly. It turns out that docker.io/library/golang:1.18 is approximately 350MB, and every one of the ENV, WORKDIR, and COPY lines below that was generating another 350MB cache image. By the time this build was done running, it was using several GB of storage space, the vast majority of it being duplicated copies of the golang container image with minor addons.

Under Kaniko (with suitable modifications to replace RUN --mount... with something that works), all of those cache layers for the ENV lines are only a few KB.

I assume this is probably an artifact of how buildah handles the layers - each layer is cumulative from the layers below it - but it produces an enormous amount of wasted cache space.

Is there any way to customize or simplify the caching so it doesn't take up so much space on the server with duplicate data?

[rhatdan]

@nalind @flouthoc PTAL

[codethief]

What is the current state of this? We are currently using Kaniko to build images in containerized CI jobs (Gitlab k8s runners) but, unfortunately, Kaniko's caching performance isn't great, so we have been looking for alternatives, and Buildah had looked quite promising, given the possibility to run it inside containers (w/o privileges) and its recent addition of --cache-to/from.

[codethief]

@kohenkatz Did you ever find a solution here?

and every one of the ENV, WORKDIR, and COPY lines below that was generating another 350MB cache image.

Which --storage-driver are you using? I'm asking because you mentioned using Buildah as a Kaniko alternative, and people often recommend using --storage-driver vfs to run buildah in such a containerized environment (at least when the container lacks permissions to use (fuse-)overlayfs mounts).

Now, vfs is known for being slow and having to copy data around for every layer. In theory I would not expect this to affect the size and contents of the image layers that end up in the cache (I would expect vfs to still diff the layers' root file systems against each other) but maybe I'm wrong?