Add ppc64le/s390x support to generate operators from the testdata samples

[gquillar]

Feature Request

Describe the problem you need a feature to resolve.

On ppc64le or s390x architectures, we have issues when trying to generate the operator-sdk Ansible/Helm testdata samples or building Ansible/Go/Helm-based operators using operator-sdk and these testdata samples:

• The kustomize binary v3.8.7 doesn't exist for ppc64le/s390x failing the make generatecommand.

$ make generate
Error generating the memcache_operator example for helm & ansible:
Command ‘make bundle IMG=quay.io/example/memcached-operator:v0.0.1’ run in testdata/helm/memcached-operator and in testdata/ansible/memcached-operator fails:
running: make bundle IMG=quay.io/example/memcached-operator:v0.0.1
time="2022-03-15T09:14:55-07:00" level=error msg="error creating the bundle: make bundle IMG=quay.io/example/memcached-operator:v0.0.1 failed with error: (exit status 2) make[1]: Entering directory '/home/gquillar/Github/operator-sdk/testdata/helm/memcached-operator'\n\ngzip: stdin: not in gzip format\ntar: Child returned status 1\ntar: Error is not recoverable: exiting now\nmake[1]: *** [Makefile:113: kustomize] Error 2\nmake[1]: Leaving directory '/home/gquillar/Github/operator-sdk/testdata/helm/memcached-operator'\n"
exit status 1
make: *** [Makefile:43: generate] Error 1

• The manager binary inside the Golang manager images is built using the amd64 arch. So the Go-based operator
  created following operator-sdk quick-start can not be installed on ppc64 or s390x cluster.

# cd testdata/go/v3/memcached-operator
# make docker-build
docker build -t controller:latest .
[1/2] STEP 1/9: FROM golang:1.17 AS builder
[1/2] STEP 2/9: WORKDIR /workspace
--> Using cache ec866082b1fb0d6ac1a1c9fa8bfed85aa63d9d9f5d098a363f8622aca647823d
--> ec866082b1f
[1/2] STEP 3/9: COPY go.mod go.mod
--> 8a4c8a242db
[1/2] STEP 4/9: COPY go.sum go.sum
--> b0ab8bd26cb
[1/2] STEP 5/9: RUN go mod download
--> a62d158aa81
[1/2] STEP 6/9: COPY main.go main.go
--> 8fcaf69e413
[1/2] STEP 7/9: COPY api/ api/
--> 567acaa4099
[1/2] STEP 8/9: COPY controllers/ controllers/
--> d8b13c7fa76
[1/2] STEP 9/9: RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -o manager main.go
--> 623ddb3e071
[2/2] STEP 1/5: FROM gcr.io/distroless/static:nonroot
[2/2] STEP 2/5: WORKDIR /
--> Using cache d7d80c0ada6f1b22a8866c93c00581c2477041e63a1b2606ee5c44a288c9f8cf
--> d7d80c0ada6
[2/2] STEP 3/5: COPY --from=builder /workspace/manager .
--> febcaac6651
[2/2] STEP 4/5: USER 65532:65532
--> 59c5696b614
[2/2] STEP 5/5: ENTRYPOINT ["/manager"]
[2/2] COMMIT controller:latest
--> f8ad1ed24df
Successfully tagged localhost/controller:latest
f8ad1ed24df5b068980339b9af0cece40039fa99b559694417e7d6959af87658
# docker run --rm -it localhost/controller:latest version    # try to run the embedded manager binary
standard_init_linux.go:228: exec user process caused: exec format error

• The opm binary v1.19.1 used to build the catalog images for the Ansible/Go/Helm samples doesn't exist for ppc64le/s390x.
  So the curl command to get the opm binary will create a bin/opm file containing 'Not Found'.

# cd testdata/ansible/memcached-operator
# make catalog-build
./bin/opm index add --container-tool docker --mode semver --tag example.com/memcached-operator-catalog:v0.0.1 --bundles example.com/memcached-operator-bundle:v0.0.1
./bin/opm: line 1: Not: command not found
make: *** [Makefile:197: catalog-build] Error 127
# ls -l bin/opm
-rwxr-xr-x 1 root root 9 May 20 05:55 bin/opm
# cat bin/opm
Not Found

Describe the solution you'd like.

Concerning the kustomize binary,

• ppc64le/s390x is supported in v4.5 but v4 is a breaking change.
• a new plugin kustomize/v2-alpha is in development to use kustomize v4 (PR-2583), but not yet available and will be an alpha version
• so to support ppc64le/s390x, the proposed solution is to build locally the kustomize binary from 3.8.7 source tarball (see PR-4612)
• since the testdata samples are generated, the changed would be done in
  kubebuilder for Go, in
  pkg/plugins/golang/v3/scaffolds/internal/templates/makefile.go and in
  operator-sdk for Ansible & Helm in
  internal/plugins/ansible/v1/scaffolds/internal/templates/makefile.go and in
  internal/plugins/helm/v1/scaffolds/internal/templates/makefile.go.

Concerning the manager binary in the Go manager image, since the the testdata Go samples are generated from kubebuilder, the change could be done in kubebuilder in pkg/plugins/golang/v3/scaffolds/internal/templates/makefile.go and pkg/plugins/golang/v3/scaffolds/internal/templates/dockerfile.go to pass the GOARCH to the Dockerfile in order to build the managerbinary with the correct arch. Currently, we have:
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -o manager main.go

Concerning the opm binary, the version could be upgraded to a version 1.20+ which supports both ppc64le and s390x.

operator-framework/operator-sdk#5674, operator-framework/operator-sdk#5736 and operator-framework/operator-sdk#5676 were initially opened to fix these issues on P and Z, but the proposed changes were wrong since they were done directly in the testdata directory of operator-sdk, which is a generated directory. So the change has to be made in kubebuider or in the operator-sdk generating the testdata content.

/language go
/language ansible
/language helm

[camilamacedo86]

Hi @gquillar,

The samples under the testdata are projects scaffolded with the code source/SDK tool. We cannot make the samples support ppc64le or s390x. Instead, we need Kubebuilder and SDK producing projects that support these architectures. Also, probably plugins that indeed have the source code outside of the SDK repo.

Note that is not only about changing the Makefile. (https://github.com/operator-framework/operator-sdk/pull/5736/files) it is also about producing images and binaries that can be used with. Please, see, for example, the task open to Kubebuilder support darwin/arm64: kubernetes-sigs/kubebuilder#1932.

In this way:

Regards the proposed changes in the Dockerfile

if you want to change the Dockerfile for further releases, then the change needs to be done in the templates,

For Golang, it must be done in: https://github.com/kubernetes-sigs/kubebuilder/blob/master/pkg/plugins/golang/v3/scaffolds/internal/templates/dockerfile.go

For Ansible/Helm and any other language plugin, we need to update the Dockerfile of the specific plugin.
In this way, I'd recommend you push a PR against Kubebuilder, and then after that gets merged, we need to update all language plugins inside of SDK and those used by it and have the code on other repos.

Regards the proposed changes to use an upper version of OPM

The PR #5676. It shows fine. I just added small nits to it.

Regards upgrade kustomize to v4

I do not think that we can move with this one: #5674
When we bump the next Kubebuilder release we will be able to check if we need to begin to use the new kustomize/v2-alpha to have the version v4. See: kubernetes-sigs/kubebuilder#2583

Also, we need to evaluate if we can make the helm/ansible plugins begin to use kustomize.v2-alpha directly or if we need new versions of these plugins as well.

[gquillar]

PR #2697 has been opened on kubernetes-sigs/kubebuilder to fix the arch used to build the binary in the go/v3 scaffold templates.

[asmacdo]

Since this is now an upstream issue, I think we are good to close here? Please reopen if necessary.