Build multi-arch image

[xunholy]

Although currently arm64 & amd64 architectures are supported it requires the --arch to alter and in a multi-arch cluster, this is also very limiting. GHCR supports multi-arch images.

I propose to move architecture-specific images and use buildx with QEMU to support more architectures. This would remove the need for --arch flag completely and allow for multi-architecture clusters without requiring to set affinity due to image limitations in their current state.

[stefanprodan]

We push both armv7 and arm64 to the same registry as we want to tract ARM usage vs AMD64. The gotk installer will pin the controllers on nodes based on the --arch argument so multi-architecture clusters are supported.

[hiddeco]

1. kustomize (the binary) does not have support for ARM64, so TARGETPLATFORM would not have brought you anywhere.
2. We want to track ARM vs AMD64 usage. Which at present is not possible without providing separate images.

This is discussion is starting to show signs of a vicious circle instead of being fruitful.

[onedr0p]

To be fair I'm okay with you guys not supporting multiarch images, I'm just a terrible debater 😅 I apologize for that.

[xunholy]

To keep things constructive @hiddeco with buildx and the appropriate approach we can help support the new experience with multi-arch images, it might mean losing that valuable piece of information but for a far better UX. Things like kustomize aren't supported because their upstream don't provide it, there are plenty of people already using it with ARM64 in many environments with almost no changes by leveraging buildx. EG. https://github.com/raspbernetes/multi-arch-images/blob/master/build/kustomize/Dockerfile

[stefanprodan]

We do publish a multi-arch image but only for ARM (armv7+arm64) and we do use buildx. docker pull ghcr.io/fluxcd/kustomize-controller-arm64:v0.1.0 works on armv7/v8. We have no intention of merging the ARM layers under the main image.

[onedr0p]

One last question, wouldn't there be an issue where if you are running a multi-arch cluster and you deployed gotk with the --arch=arm64 flag, now imagine a scenario where you need to take your arm nodes offline for whatever reason. The deployments would stop working, correct?

The fix it seems would be to rerun gotk bootstrap without the --arch flag, but that requires manual intervention.

[onedr0p]

I agree with @xunholy , having one image would simplify builds and ease of use.

[stefanprodan]

What builds are those?

[onedr0p]

The CI pipelines, I've been building multiarch images for awhile now and the process is pretty streamlined.

[stefanprodan]

You don't have to build GOTK we do that for you.

[onedr0p]

I know, I'm talking about your own maintaince of building images and reducing lines of code, there and in gotk source :)

But if you want to really track usage of separate arch, there's no way to do that with a multiarch image.

[billimek]

Multi-arch images would be useful for kubernetes clusters operating in mixed-architecture environments. Having separate arch-specific images or tags doesn't allow for such operation.

[onedr0p]

@billimek as mentioned by @stefanprodan this is a non-issue since the gotk deployment controllers are attracted to whatever you pass into the --arch flag. I do see this as being confusing at first and not great in terms of UX. Removing the need for this flag and building multi-arch images would be 10x better for UX.

[stefanprodan]

To sum this up as I feel it is going nowhere:

• we do support the following architectures: AMD64, ARM64, ARMv7
• we do publish multi-arch images for both ARM64 and ARMv7 under ghcr.io/fluxcd/<controller>-arm64:semver
• the gotk installer works on mixed-architecture clusters, you need to specify the arch you want to pin the controllers to with gotk install --arch=amd64|arm64|arm
• we want to track ARM usage so we have no intention of merging all layers under a single image