multi-architecture builds and publish #1681

jon-nfc · 2024-01-08T11:45:30Z

SUMMARY

Add changes related to multi-architecture builds and publishing.

The desired endstate is that there is officially built and supported multi-architecture container images that can be built from a single command make docker-buildx and that on quay.io/ansible/awx-operator those same muli-architecture images are available for consumption.

ISSUE TYPE

New or Enhanced Feature

ADDITIONAL INFORMATION

Current release runbook can be found in docs/contributors-guide/release-process.md

This PR does not have the intention of changing the release workflow. The intent is that as part of the established workflow, that multi-architecture containers are built and published for use by the community.

Adjustment was made to ensure that there is ever only one image built for a given git commit and is used for the entire workflow. This prevents the scenario where it's as if a witch has casted a spell upon you, when at the end of the day there were two or more images that exist for the same commit and you were the poor sole who can't figure out what is wrong because the image you have is "Good to go!!" Bottom line no one wants to debug black magic (be in a scenario where two images exist for the same code but are different enough to make you pull your hair out). As each commit is stored on GHCR, it becomes the single source of truth for any containers built, especially for debugging purposes.

Changes to Github Actions

devel push to devel branch builds multi-arch containers and tags it with the git commit and stores it on GHCR in addition to the existing step of pushing said manifest to quay.io with the devel tag
stage when triggered no longer builds it's own image. pulls the image from GHCR built in devel action and follows the existing flow of tagging with the specified version and pushing back to GHCR
promote Github Release No change, however now uses the multi-arch manifest.

Other Changes

reference: ansible/eda-server-operator#158

makefile now has a target docker-buildx this target provides the basis for all multi-arch container builds. As buildx creates a manifest with all container architectures included, as part of the same command will automagic push the image to the tagged registry. This command can also be used in development
- from a terminal make docker-buildx
- To customize the image name and registry IMG=registry/name/name2:tag make docker-buildx.
- To specifiy different architectures to build PLATFORMS=linux/arm64,linux/amd64 make docker-buildx
- multiple variables can be addded to the command should it be desired.

Fixes issues

jon-nfc · 2024-01-08T13:08:44Z

awx-operator/.github/workflows/publish-operator-hub.yaml

Lines 78 to 86 in 07427be

    
                 - name: Build and publish bundle to operator-hub 
        
                   working-directory: awx-operator-${{ env.VERSION }} 
        
                   env: 
        
                     IMG_REPOSITORY: ${{ env.IMAGE_REGISTRY }}/${{ env.IMAGE_REGISTRY_ORGANIZATION }} 
        
                     GITHUB_TOKEN: ${{ secrets.AWX_AUTO_GITHUB_TOKEN }} 
        
                   run: | 
        
                     git config --global user.email "awx-automation@redhat.com" 
        
                     git config --global user.name "AWX Automation" 
        
                     ./hack/publish-to-operator-hub.sh

which leads to

awx-operator/hack/publish-to-operator-hub.sh

Lines 43 to 45 in 07427be

    
           # Build bundle and catalog images 
        
           make bundle-build bundle-push BUNDLE_IMG=$BUNDLE_IMG IMG=$OPERATOR_IMG 
        
           make catalog-build catalog-push CATALOG_IMG=$CATALOG_IMG BUNDLE_IMGS=$BUNDLE_IMG BUNDLE_IMG=$BUNDLE_IMG IMG=$OPERATOR_IMG

As part of the publish there is another build and push of the image. Can this be safely removed? AS the complete manifest is already pushed to quay.io in the promote action

awx-operator/Makefile

Lines 235 to 237 in 07427be

    
           .PHONY: catalog-build 
        
           catalog-build: opm ## Build a catalog image. 
        
           	$(OPM) index add --container-tool ${CONTAINER_CMD} --mode semver --tag $(CATALOG_IMG) --bundles $(BUNDLE_IMGS) $(FROM_INDEX_OPT)

Also I noticed that step make catalog-build uses something called opm, what is it and in it's current usage, can it handle docker manifests?

jon-nfc · 2024-01-08T13:52:36Z

Don't know who to ping, PR ready for review.

also require someone in the know to answer my above question

rooftopcellist · 2024-01-08T15:46:15Z

@jon-nfc

As part of the publish there is another build and push of the image. Can this be safely removed? AS the complete manifest is already pushed to quay.io in the promote action

The publish-to-operator-hub.sh script is for adding the awx-operator version to https://operatorhub.io/operator/awx-operator so it can be installed with OLM. The bundle image is needed for OLM installs, it contains content from the manifests and crd directories (critically, the CSV).

bundle image build - This image is separate from the operator image. It is necessary for OLM installs, but shouldn't be any trouble for multi-arch.

Bundle images are not architecture-specific. They contain only plaintext Kubernetes manifests and operator metadata. (Reference)[https://sdk.operatorframework.io/docs/advanced-topics/multi-arch/#operator-lifecycle-manager]

catalog image build - not strictly necessary, but it is a nice-to-have because it allows for easier testing of OLM-specific parameter changes. Also handy for demos of new features because there can sometimes be a 1-2 day lag for the operator getting published to OperatorHub.io after a release.

The opm tool is a cli tool for inspecting and managing catalog images. More information can be found here.

A catalog image is a 3rd image that contains pointers to operator bundle versions it knows about, which populates the list of available operators that can be installed on Openshift or an OLM enabled k8s cluster. The newer catalog images are just plaintext on the inside as far as I know (file-based-catalogs) as far as I know.

This is not strictly needed for users as the folks from https://github.com/k8s-operatorhub/community-operators maintain a catalog image that gets installed as part of OLM when you install an operator from OperatorHub.io. So I don't think we need to worry about multi-arch for these images.

rooftopcellist · 2024-01-08T16:53:18Z

@jon-nfc it looks like there are a few linting issues that need to be cleaned up.

./.github/workflows/promote.yaml
  Warning: 32:76 [comments] too few spaces before comment
  Warning: 40:76 [comments] too few spaces before comment
  Error: 49:171 [line-length] line too long (176 > 170 characters)
  Error: 54:171 [line-length] line too long (171 > 170 characters)

./.github/workflows/stage.yml
  Warning: 58:76 [comments] too few spaces before comment
  Error: 67:171 [line-length] line too long (216 > 170 characters)

./.github/workflows/devel.yaml
  Warning: 17:76 [comments] too few spaces before comment
  Warning: 25:76 [comments] too few spaces before comment
  Error: [39](https://github.com/ansible/awx-operator/actions/runs/7447379325/job/20266238173?pr=1681#step:6:43):171 [line-length] line too long (176 > 170 characters)
  Error: [40](https://github.com/ansible/awx-operator/actions/runs/7447379325/job/20266238173?pr=1681#step:6:44):1 [empty-lines] too many blank lines (1 > 0)

.github/workflows/stage.yml

As there are ENV vars set by the build process, there needs to be a way of passing them at build time PR ansible#1681

As part of the release process, env vars for the operator and awx must be set. As such the image needs to be rebuilt. PR ansible#1681

PR ansible#1681

.github/workflows/stage.yml

jon-nfc · 2024-01-09T07:04:20Z

This is not strictly needed for users as the folks from https://github.com/k8s-operatorhub/community-operators maintain a catalog image that gets installed as part of OLM when you install an operator from OperatorHub.io. So I don't think we need to worry about multi-arch for these images.

I though as much.

rooftopcellist · 2024-01-10T03:06:32Z

@jon-nfc I think this is a good approach. I didn't spot any issues while reading this through except for one more linting error CI is complaining about. I would like to run through a mock release on my fork first though before merging. I'll plan on doing that tomorrow.

./.github/workflows/devel.yaml
  Error: 42:1 [empty-lines] too many blank lines (1 > 0)

cc @TheRealHaoLiu for review as well.

42:1 [empty-lines] too many blank lines (1 > 0) PR ansible#1681

jon-nfc · 2024-01-10T04:55:08Z

@jon-nfc I think this is a good approach. I didn't spot any issues while reading this through except for one more linting error CI is complaining about. I would like to run through a mock release on my fork first though before merging. I'll plan on doing that tomorrow.
./.github/workflows/devel.yaml
  Error: 42:1 [empty-lines] too many blank lines (1 > 0)
cc @TheRealHaoLiu for review as well.

Righto, I'll have everything good to go in the next few hours.

tomorrow

depends on where you are, where i am it could still be yesterday for you.

rooftopcellist · 2024-01-10T21:28:33Z

.github/workflows/stage.yml

-          IMAGE_TAG_BASE=ghcr.io/${{ github.repository_owner }}/awx-operator \
-          VERSION=${{ github.event.inputs.version }} make docker-build docker-push
+              --build-arg OPERATOR_VERSION=${{ github.event.inputs.version }}" \
+          IMG=ghcr.io/${{ github.repository_owner }}/${{ github.repository }}:${{ github.event.inputs.version }} \


Suggested change

IMG=ghcr.io/${{ github.repository_owner }}/${{ github.repository }}:${{ github.event.inputs.version }} \

IMG=ghcr.io/${{ github.repository }}:${{ github.event.inputs.version }} \

We need to remove ${{ github.repository_owner }} here because it is already there from when it is set during the checkout:

https://github.com/ansible/awx-operator/pull/1681/files#diff-ca6f66a88a8c14e7412271d092d23b124f23052eb38134788e5a992f0522f574R49

awx-operator/.github/workflows/stage.yml

Lines 46 to 50 in 4d79365

- name: Checkout awx-operator

uses: actions/checkout@v3

with:

repository: ${{ github.repository_owner }}/awx-operator

path: awx-operator

I did not know this, however note taken.

question, re variable preceedence

awx-operator/.github/workflows/stage.yml

Lines 40 to 50 in 4d79365

- name: Checkout awx

uses: actions/checkout@v3

with:

repository: ${{ github.repository_owner }}/awx

path: awx

- name: Checkout awx-operator

uses: actions/checkout@v3

with:

repository: ${{ github.repository_owner }}/awx-operator

path: awx-operator

there are two checkouts, which variable takes precedence? first set, last set?

.github/workflows/devel.yaml

molecule (--skip-tags=replicas): .github/workflows/devel.yaml#L41 41:65 [new-line-at-end-of-file] no new line character at the end of file PR ansible#1681

jon-nfc · 2024-01-11T08:29:36Z

.github/workflows/devel.yaml

+        run: |
+          docker buildx imagetools create \
+            ghcr.io/${{ github.repository_owner }}/${{ github.repository }}:${{ github.sha }} \
+            --tag quay.io/ansible/${{ github.repository }}:devel


Don't know if there is an issue with the linter but it inconsistent.

466b0cf too many blank lines 0>1 when there were two blank lines https://github.com/ansible/awx-operator/actions/runs/7457722546

4d79365 no blank line at end of file no new line character at the end of file https://github.com/ansible/awx-operator/actions/runs/7470471612

A default set of platforms is specified and will build those if var `PLATFORMS` is not specified on the CLI Issue ansible#31 fixes ansible#1680

@ghcr

1. builds containers and stores @ghcr with tag that is git commit. 2. pushes the image to quay.io/ansible with tag devel. Issue ansible#31

It's not good practice to rebuild images as the scenario exists where the same code base builds different images. As such only use images from a single build point. 1. Pulls the image that matches the git commit 2. Tag image with the specified version and push to ghcr Issue ansible#31

…i-arch manifest 1. Pulls the complete manifest containing all architectures that matches the version from ghcr 2. Tag manifest with the specified version and publish to quay.io 3. Tag manifest as 'latest' and publish to quay.io fixes ansible#31

As there are ENV vars set by the build process, there needs to be a way of passing them at build time PR ansible#1681

As part of the release process, env vars for the operator and awx must be set. As such the image needs to be rebuilt. PR ansible#1681

PR ansible#1681

42:1 [empty-lines] too many blank lines (1 > 0) PR ansible#1681

molecule (--skip-tags=replicas): .github/workflows/devel.yaml#L41 41:65 [new-line-at-end-of-file] no new line character at the end of file PR ansible#1681

jon-nfc · 2024-01-16T01:49:52Z

branch updated with rebase.

PR ready for review

TheRealHaoLiu · 2024-01-16T16:28:42Z

.github/workflows/promote.yaml

+            ghcr.io/${{ github.repository_owner }}/${{ github.repository }}:${TAG_NAME} \
+            --tag quay.io/ansible/${{ github.repository }}:${TAG_NAME}


based on https://docs.github.com/en/enterprise-cloud@latest/actions/learn-github-actions/variables#default-environment-variables

GITHUB_REPOSITORY | The owner and repository name. For example, octocat/Hello-World.

i think we need to take in a variable for "which quay repo to push this to"

@TheRealHaoLiu

i have an outstanding question above re this from the review that @rooftopcellist did. That's why the discussions are not resolved as the next reviewer can see and resolve as completed and as appropriate.

#1681 (comment). this question still remains!!!!

.github/workflows/promote.yaml

.github/workflows/devel.yaml

rooftopcellist · 2024-01-16T21:00:19Z

@jon-nfc ,

@TheRealHaoLiu and I tested out this workflow on our fork today and got it working with the changes in this PR.

Mainly, we modified how it runs so that we could test it on a fork and push the images to our personal quay registry so as not to affect the official awx-operator images while testing.

There were also a few variable refs that needed to be changed for the workflow to work. Notably:

ghcr.io/${{ github.repository_owner }}/${{ github.repository }} evaluates to ghcr.io/ansible/ansible/awx-operator, which failed.

I also changed how it creates the draft release so that it uses the github action for it rather than custom logic to git the GitHub API.

I opened a PR to your branch here:

Modify Release GHA to run on fork jon-nfc/awx-operator#1

Note that a variable needs to be created in the GitHub repo if you want to override the default when triggering the promote pipeline by publishing the draft release:

Here is the successful test release on my fork:

Modify Release GHA to run on fork

github-actions bot added the community label Jan 8, 2024

jon-nfc force-pushed the multi-arch-builds branch 2 times, most recently from 4935911 to 506369a Compare January 8, 2024 12:16

jon-nfc marked this pull request as ready for review January 8, 2024 13:48

jon-nfc mentioned this pull request Jan 8, 2024

Add Release Workflow and Single Command Install artifact ansible/eda-server-operator#159

Merged

rooftopcellist requested review from TheRealHaoLiu and rooftopcellist January 8, 2024 14:59

rooftopcellist reviewed Jan 8, 2024

View reviewed changes

.github/workflows/stage.yml Show resolved Hide resolved

jon-nfc added a commit to jon-nfc/awx-operator that referenced this pull request Jan 9, 2024

fix(makefile): build args required for the container build

78317ff

As there are ENV vars set by the build process, there needs to be a way of passing them at build time PR ansible#1681

jon-nfc added a commit to jon-nfc/awx-operator that referenced this pull request Jan 9, 2024

fix(workflow): container requires that ENV vars be set.

1a94049

As part of the release process, env vars for the operator and awx must be set. As such the image needs to be rebuilt. PR ansible#1681

jon-nfc added a commit to jon-nfc/awx-operator that referenced this pull request Jan 9, 2024

chore: correct liniting errorr

e4c1c65

PR ansible#1681

jon-nfc commented Jan 9, 2024

View reviewed changes

.github/workflows/stage.yml Show resolved Hide resolved

.github/workflows/stage.yml Show resolved Hide resolved

This was referenced Jan 9, 2024

Pin API and UI image refs in released operator images ansible/eda-server-operator#125

Closed

ARM64 images built and Available on AWX repository ansible/awx#14643

Closed

jon-nfc requested a review from rooftopcellist January 9, 2024 14:02

jon-nfc added a commit to jon-nfc/awx-operator that referenced this pull request Jan 10, 2024

chore: correct linting error

466b0cf

42:1 [empty-lines] too many blank lines (1 > 0) PR ansible#1681

rooftopcellist reviewed Jan 10, 2024

View reviewed changes

rooftopcellist reviewed Jan 11, 2024

View reviewed changes

.github/workflows/devel.yaml Outdated Show resolved Hide resolved

jon-nfc added a commit to jon-nfc/awx-operator that referenced this pull request Jan 11, 2024

chore: correct linting error

6d0ba09

molecule (--skip-tags=replicas): .github/workflows/devel.yaml#L41 41:65 [new-line-at-end-of-file] no new line character at the end of file PR ansible#1681

jon-nfc added a commit to jon-nfc/awx-operator that referenced this pull request Jan 11, 2024

chore: correct linting error

4d79365

molecule (--skip-tags=replicas): .github/workflows/devel.yaml#L41 41:65 [new-line-at-end-of-file] no new line character at the end of file PR ansible#1681

jon-nfc force-pushed the multi-arch-builds branch from 6d0ba09 to 4d79365 Compare January 11, 2024 08:12

jon-nfc commented Jan 11, 2024

View reviewed changes

Add multi-arch image builds to makefile

4f8999c

A default set of platforms is specified and will build those if var `PLATFORMS` is not specified on the CLI Issue ansible#31 fixes ansible#1680

jon-nfc added 8 commits January 16, 2024 11:18

feat(workflow): update devel workflow to build multi-arch containers.

02c1112

1. builds containers and stores @ghcr with tag that is git commit. 2. pushes the image to quay.io/ansible with tag devel. Issue ansible#31

fix(makefile): build args required for the container build

167ff9d

As there are ENV vars set by the build process, there needs to be a way of passing them at build time PR ansible#1681

fix(workflow): container requires that ENV vars be set.

36fee78

As part of the release process, env vars for the operator and awx must be set. As such the image needs to be rebuilt. PR ansible#1681

chore: correct liniting errorr

07bb49e

PR ansible#1681

chore: correct linting error

82e31aa

42:1 [empty-lines] too many blank lines (1 > 0) PR ansible#1681

chore: correct linting error

9442e0a

molecule (--skip-tags=replicas): .github/workflows/devel.yaml#L41 41:65 [new-line-at-end-of-file] no new line character at the end of file PR ansible#1681

jon-nfc force-pushed the multi-arch-builds branch from 4d79365 to 9442e0a Compare January 16, 2024 01:48

TheRealHaoLiu reviewed Jan 16, 2024

View reviewed changes

.github/workflows/promote.yaml Outdated Show resolved Hide resolved

TheRealHaoLiu reviewed Jan 16, 2024

View reviewed changes

.github/workflows/devel.yaml Outdated Show resolved Hide resolved

jon-nfc requested a review from TheRealHaoLiu January 16, 2024 16:36

rooftopcellist and others added 2 commits January 16, 2024 16:07

Modify Release GHA to run on fork

a60fd23

Merge pull request #1 from rooftopcellist/multi-arch-builds-test

da0bac0

Modify Release GHA to run on fork

rooftopcellist mentioned this pull request Jan 16, 2024

Multi-arch Operator Release Changes ansible/awx#14772

Merged

rooftopcellist enabled auto-merge (squash) January 16, 2024 22:00

rooftopcellist approved these changes Jan 16, 2024

View reviewed changes

rooftopcellist merged commit 7e2c2bf into ansible:devel Jan 16, 2024
6 checks passed

jon-nfc mentioned this pull request Jan 19, 2024

Add ARM Support to the Operator #31

Closed

jon-nfc mentioned this pull request Feb 14, 2024

multi-architecture builds only occur for devel tag #1711

Open

3 tasks

rooftopcellist mentioned this pull request Mar 7, 2024

"make deploy"-command doesn't work on arm64 #628

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

multi-architecture builds and publish #1681

multi-architecture builds and publish #1681

jon-nfc commented Jan 8, 2024 •

edited

Loading

jon-nfc commented Jan 8, 2024

jon-nfc commented Jan 8, 2024

rooftopcellist commented Jan 8, 2024

rooftopcellist commented Jan 8, 2024

jon-nfc commented Jan 9, 2024

rooftopcellist commented Jan 10, 2024 •

edited

Loading

jon-nfc commented Jan 10, 2024

rooftopcellist Jan 10, 2024

jon-nfc Jan 11, 2024

jon-nfc Jan 11, 2024

jon-nfc commented Jan 16, 2024

TheRealHaoLiu Jan 16, 2024

jon-nfc Jan 16, 2024

rooftopcellist commented Jan 16, 2024

	IMG=ghcr.io/${{ github.repository_owner }}/${{ github.repository }}:${{ github.event.inputs.version }} \
	IMG=ghcr.io/${{ github.repository }}:${{ github.event.inputs.version }} \

	- name: Checkout awx-operator
	uses: actions/checkout@v3
	with:
	repository: ${{ github.repository_owner }}/awx-operator
	path: awx-operator

	- name: Checkout awx
	uses: actions/checkout@v3
	with:
	repository: ${{ github.repository_owner }}/awx
	path: awx

	- name: Checkout awx-operator
	uses: actions/checkout@v3
	with:
	repository: ${{ github.repository_owner }}/awx-operator
	path: awx-operator

		ghcr.io/${{ github.repository_owner }}/${{ github.repository }}:${TAG_NAME} \
		--tag quay.io/ansible/${{ github.repository }}:${TAG_NAME}

multi-architecture builds and publish #1681

multi-architecture builds and publish #1681

Conversation

jon-nfc commented Jan 8, 2024 • edited Loading

SUMMARY

ISSUE TYPE

ADDITIONAL INFORMATION

Changes to Github Actions

Other Changes

Fixes issues

jon-nfc commented Jan 8, 2024

jon-nfc commented Jan 8, 2024

rooftopcellist commented Jan 8, 2024

rooftopcellist commented Jan 8, 2024

jon-nfc commented Jan 9, 2024

rooftopcellist commented Jan 10, 2024 • edited Loading

jon-nfc commented Jan 10, 2024

rooftopcellist Jan 10, 2024

Choose a reason for hiding this comment

jon-nfc Jan 11, 2024

Choose a reason for hiding this comment

jon-nfc Jan 11, 2024

Choose a reason for hiding this comment

jon-nfc commented Jan 16, 2024

TheRealHaoLiu Jan 16, 2024

Choose a reason for hiding this comment

jon-nfc Jan 16, 2024

Choose a reason for hiding this comment

rooftopcellist commented Jan 16, 2024

jon-nfc commented Jan 8, 2024 •

edited

Loading

rooftopcellist commented Jan 10, 2024 •

edited

Loading