✨ add options and guidance to provide solutions for other platforms and within multiple architecture support

[camilamacedo86]

Description

• Add a new OPTIONAL option to build images with multi-cluster support. (make docker-buildx-push)

With the new target introduced in this PR is possible to build the manager image by providing support to multi-platforms. Also, the changes made in the Dockerfile allow users to use other options and help them create customizations to achieve this goal indeed with other tools.

• Add a doc providing the explanations and guiding operator authors
• Change the kustomize scaffolds to use node Affinity by default so that they can also comply with the recommendations
• Change the deploy-image/v1-alpha plugin in order to provide the setup for node affinity in the controller reconciliation to build the Deployment used to run the Operand image
• Fix the issue to call docker-build from platforms / hosts that will not provide BUILPLATFORM == linux/amd64
  (before this PR we have authors building solutions where the container will have the support for Linux/arm64 and the binary to run the manager was built with linux/amd64 because it was fixed )

Motivation

• I am an operator author I'd like to build a project that can be distributed and work no matter the platform.
• I am an operator author I'd like to customize my project to be distributed to another specific platform

Closes: #2697
Closes: #2850

Local Test with new OPTIONAL target make docker-buildx:

Building the image:

$ make docker-buildx IMG=docker.io/cmacedo/testkube:0.0.3
docker buildx create --name project-v3-builder
project-v3-builder
docker buildx use project-v3-builder
docker buildx build --push --platform=linux/arm64,linux/amd64,linux/s390x,linux/ppc64le --tag docker.io/cmacedo/testkube:0.0.3 .
[+] Building 809.2s (44/44) FINISHED                                                                               
 => [internal] booting buildkit                                                                               4.5s
 => => pulling image moby/buildkit:buildx-stable-1                                                            2.1s
 => => creating container buildx_buildkit_project-v3-builder0                                                 2.3s
 => [internal] load .dockerignore                                                                             0.1s
 => => transferring context: 171B                                                                             0.0s
 => [internal] load build definition from Dockerfile                                                          0.1s
 => => transferring dockerfile: 1.35kB                                                                        0.0s
 => [linux/s390x internal] load metadata for gcr.io/distroless/static:nonroot                                 2.0s
 => [linux/amd64 internal] load metadata for docker.io/library/golang:1.18                                    2.1s
 => [linux/amd64 internal] load metadata for gcr.io/distroless/static:nonroot                                 2.0s
 => [linux/arm64 internal] load metadata for gcr.io/distroless/static:nonroot                                 2.0s
 => [linux/ppc64le internal] load metadata for gcr.io/distroless/static:nonroot                               2.0s
 => [auth] library/golang:pull token for registry-1.docker.io                                                 0.0s
 => [linux/amd64 builder 1/9] FROM docker.io/library/golang:1.18@sha256:f1741ed3855b1a558f3c057f777667a4b73f  0.0s
 => => resolve docker.io/library/golang:1.18@sha256:f1741ed3855b1a558f3c057f777667a4b73ff250c4f1bee7f0388cc2  0.1s
 => => sha256:06ef4a250537829c59bb616d2c589fde2f8780ba55d21e5684ebd3ce1193ee30 157B / 157B                    0.2s
 => => sha256:95edc8028182acfa80667fa6a3b869f79574821298f786228e873ac10b3dd7bd 141.88MB / 141.88MB           36.1s
 => => sha256:dacb042d4b557bba852cbd15b9e3251e6b11f0b755b89541540c1894b7c4ef2c 85.91MB / 85.91MB             33.9s
 => => sha256:53ad072f9cd16fc8eb93b182b20e758e11acc0ef60babef0bf1043c08de1901a 54.58MB / 54.58MB             17.8s
 => => sha256:3e94d13e55e7a4ef17ff21376f57fb95c7e1706931f8704aa99260968d81f6e4 5.16MB / 5.16MB               12.4s
 => => sha256:fa9c7528c685216129e8e67bf362a7702e7b1daa585ab85546a41508830657d6 10.88MB / 10.88MB              5.1s
 => => sha256:1671565cc8df8c365c9b661d3fbc164e73d01f1b0430c6179588428f99a9da2e 55.01MB / 55.01MB             16.2s
 => => extracting sha256:1671565cc8df8c365c9b661d3fbc164e73d01f1b0430c6179588428f99a9da2e                     5.6s
 => => extracting sha256:3e94d13e55e7a4ef17ff21376f57fb95c7e1706931f8704aa99260968d81f6e4                     0.5s
 => => extracting sha256:fa9c7528c685216129e8e67bf362a7702e7b1daa585ab85546a41508830657d6                     0.5s
 => => extracting sha256:53ad072f9cd16fc8eb93b182b20e758e11acc0ef60babef0bf1043c08de1901a                     4.6s
 => => extracting sha256:dacb042d4b557bba852cbd15b9e3251e6b11f0b755b89541540c1894b7c4ef2c                     4.4s
 => => extracting sha256:95edc8028182acfa80667fa6a3b869f79574821298f786228e873ac10b3dd7bd                    10.9s
 => => extracting sha256:06ef4a250537829c59bb616d2c589fde2f8780ba55d21e5684ebd3ce1193ee30                     0.0s
 => [linux/amd64 stage-1 1/3] FROM gcr.io/distroless/static:nonroot@sha256:1f580b0a1922c3e54ae15b0758b5747b2  1.0s
 => => resolve gcr.io/distroless/static:nonroot@sha256:1f580b0a1922c3e54ae15b0758b5747b260bd99d39d40c2edb3e7  0.1s
 => => sha256:0a602d5f6ca3de9b0e0d4d64e8857e504ec7a8c47f1ec617d82a81f6c64b0fe8 801.01kB / 801.01kB            0.4s
 => => extracting sha256:0a602d5f6ca3de9b0e0d4d64e8857e504ec7a8c47f1ec617d82a81f6c64b0fe8                     1.0s
 => [internal] load build context                                                                             0.2s
 => => transferring context: 142.22kB                                                                         0.0s
 => [linux/s390x stage-1 1/3] FROM gcr.io/distroless/static:nonroot@sha256:1f580b0a1922c3e54ae15b0758b5747b2  1.0s
 => => resolve gcr.io/distroless/static:nonroot@sha256:1f580b0a1922c3e54ae15b0758b5747b260bd99d39d40c2edb3e7  0.1s
 => => sha256:e4d4f1812cc432feba11395231ddd13bc5385bf010a4acbe41c62ea81091b656 801.01kB / 801.01kB            1.7s
 => => extracting sha256:e4d4f1812cc432feba11395231ddd13bc5385bf010a4acbe41c62ea81091b656                     1.0s
 => [linux/arm64 stage-1 1/3] FROM gcr.io/distroless/static:nonroot@sha256:1f580b0a1922c3e54ae15b0758b5747b2  1.1s
 => => resolve gcr.io/distroless/static:nonroot@sha256:1f580b0a1922c3e54ae15b0758b5747b260bd99d39d40c2edb3e7  0.1s
 => => sha256:a6f41b961ab0c6744fe9133b4fc2df373903f18ee5bf90f4ca00ff1a8c989173 801.01kB / 801.01kB            1.6s
 => => extracting sha256:a6f41b961ab0c6744fe9133b4fc2df373903f18ee5bf90f4ca00ff1a8c989173                     1.1s
 => [linux/ppc64le stage-1 1/3] FROM gcr.io/distroless/static:nonroot@sha256:1f580b0a1922c3e54ae15b0758b5747  1.0s
 => => resolve gcr.io/distroless/static:nonroot@sha256:1f580b0a1922c3e54ae15b0758b5747b260bd99d39d40c2edb3e7  0.1s
 => => sha256:d80414a7217c1c756522dad11d0bba172ba8458b4a5ad4debdc0d697fdbd206e 801.01kB / 801.01kB            2.3s
 => => extracting sha256:d80414a7217c1c756522dad11d0bba172ba8458b4a5ad4debdc0d697fdbd206e                     1.0s
 => [linux/amd64 builder 2/9] WORKDIR /workspace                                                              0.8s
 => [linux/amd64 builder 3/9] COPY go.mod go.mod                                                              0.0s
 => [linux/amd64 builder 4/9] COPY go.sum go.sum                                                              0.0s
 => [linux/amd64->s390x builder 5/9] RUN go mod download                                                     33.4s
 => [linux/amd64->ppc64le builder 5/9] RUN go mod download                                                   33.6s
 => [linux/amd64 builder 5/9] RUN go mod download                                                            36.4s
 => [linux/amd64->arm64 builder 5/9] RUN go mod download                                                     37.7s
 => [linux/amd64->s390x builder 6/9] COPY main.go main.go                                                     4.4s
 => [linux/amd64->ppc64le builder 6/9] COPY main.go main.go                                                   4.8s
 => [linux/amd64 builder 6/9] COPY main.go main.go                                                            2.0s
 => [linux/amd64->arm64 builder 6/9] COPY main.go main.go                                                     0.7s
 => [linux/amd64->s390x builder 7/9] COPY api/ api/                                                           0.5s
 => [linux/amd64->ppc64le builder 7/9] COPY api/ api/                                                         0.2s
 => [linux/amd64 builder 7/9] COPY api/ api/                                                                  0.3s
 => [linux/amd64->arm64 builder 7/9] COPY api/ api/                                                           0.2s
 => [linux/amd64->s390x builder 8/9] COPY controllers/ controllers/                                           0.2s
 => [linux/amd64->ppc64le builder 8/9] COPY controllers/ controllers/                                         0.4s
 => [linux/amd64->s390x builder 9/9] RUN CGO_ENABLED=0 GOOS=linux GOARCH=s390x go build -a -o manager main  640.4s
 => [linux/amd64->arm64 builder 8/9] COPY controllers/ controllers/                                           0.4s
 => [linux/amd64 builder 8/9] COPY controllers/ controllers/                                                  0.4s
 => [linux/amd64->ppc64le builder 9/9] RUN CGO_ENABLED=0 GOOS=linux GOARCH=ppc64le go build -a -o manager   652.8s
 => [linux/amd64 builder 9/9] RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -o manager main.go      653.2s
 => [linux/amd64->arm64 builder 9/9] RUN CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -a -o manager main  655.5s
 => [linux/s390x stage-1 2/3] COPY --from=builder /workspace/manager .                                        2.5s
 => [linux/ppc64le stage-1 2/3] COPY --from=builder /workspace/manager .                                      0.5s
 => [linux/amd64 stage-1 2/3] COPY --from=builder /workspace/manager .                                        0.5s
 => [linux/arm64 stage-1 2/3] COPY --from=builder /workspace/manager .                                        0.3s
 => exporting to image                                                                                       45.4s
 => => exporting layers                                                                                       7.4s
 => => exporting manifest sha256:260ae0740c97da01a0e61f95b4a6767bee20b3af6b0cab7a677ea6349c8f1fab             0.0s
 => => exporting config sha256:0e50ae208b83e668bc229311d024ef95adbf526f0ea252c52ebbf172585f6ed8               0.0s
 => => exporting manifest sha256:fb37eb73270f7d49e9f84278d1db2b1b9e7e1a3ff5786292c446361250fa401b             0.0s
 => => exporting config sha256:0404cdd2cb316cc2fd93e8cb72e9f517dc416d198d179d678bed4896b6566a79               0.0s
 => => exporting manifest sha256:c4ca8824f7039cc9a7c622bbd3a81b3a514ad5f1987b51c437f2fbd7341bb0cf             0.0s
 => => exporting config sha256:351bf89196c4a16581def71d383acf275b08b7650b3bd8d745864fb28166a338               0.0s
 => => exporting manifest sha256:6607f2f60d16a58f2f6117d034abe8d433c0e1ab903a5c211ed09823bd25a96f             0.0s
 => => exporting config sha256:b7fe530b3df3046f01e5bfff0c02da70fc33ecd87e9db961d87b3e598a31adf7               0.0s
 => => exporting manifest list sha256:969a82c605995d551b6791b38efdc716fa2755540be92520c00b9908a4dbd0b3        0.0s
 => => pushing layers                                                                                        35.8s
 => => pushing manifest for docker.io/cmacedo/testkube:0.0.3@sha256:969a82c605995d551b6791b38efdc716fa275554  1.9s
 => [auth] cmacedo/testkube:pull,push token for registry-1.docker.io                                          0.0s
docker buildx rm project-v3-builder

Checking the manifest list:

$ docker manifest inspect docker.io/cmacedo/testkube:0.0.3
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 739,
         "digest": "sha256:260ae0740c97da01a0e61f95b4a6767bee20b3af6b0cab7a677ea6349c8f1fab",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 739,
         "digest": "sha256:fb37eb73270f7d49e9f84278d1db2b1b9e7e1a3ff5786292c446361250fa401b",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 739,
         "digest": "sha256:c4ca8824f7039cc9a7c622bbd3a81b3a514ad5f1987b51c437f2fbd7341bb0cf",
         "platform": {
            "architecture": "s390x",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 739,
         "digest": "sha256:6607f2f60d16a58f2f6117d034abe8d433c0e1ab903a5c211ed09823bd25a96f",
         "platform": {
            "architecture": "ppc64le",
            "os": "linux"
         }
      }
   ]
}

[camilamacedo86]

@aleskandro @gquillar

[camilamacedo86]

/retest-required

[camilamacedo86]

/hold cancel

Since I get via msg the lgtm from folks as well
And it seems good enough to move forward.

[aleskandro]

lgtm

[jaypoulz]

/lgtm

Just to make official feedback I gave verbally.

[k8s-ci-robot]

@jaypoulz: changing LGTM is restricted to collaborators

In response to this:

/lgtm

Just to make official feedback I gave verbally.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[jeffdyoung]

/lgtm

[k8s-ci-robot]

@jeffdyoung: changing LGTM is restricted to collaborators

In response to this:

/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Kavinjsir]

Looks good to me!

[everettraven]

Have some nits for the doc, other than that everything else looks good to me!

[varshaprasad96]

Just nit, rest looks good! Thanks @camilamacedo86
/lgtm

[varshaprasad96]

Nit:

Suggested change

	To provide support on specific or multiple platforms, you must ensure that all images used in workloads are built to
	support the desired platforms. Note that may not be the same as the platform where you develop your solutions
	and use KubeBuilder, but instead the platform(s) where your solution should run and be distributed.
	It is recommended to build solutions that work on multiple platforms so that your project works
	on any Kubernetes cluster regardless of the underlying operating system and architecture.
	To provide support on specific or multiple platforms, you must ensure that all images used in workloads are built to
	support the desired platforms. Note that this may not be the same as the platform where you develop your solutions using KubeBuilder, but instead the platform(s) where your solution would run and will be distributed.
	It is recommended to build solutions that work on multiple platforms so that your project works
	on any Kubernetes cluster regardless of the underlying operating system and architecture.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aleskandro, camilamacedo86, jaypoulz, Kavinjsir, varshaprasad96

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [camilamacedo86,varshaprasad96]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment