Modify action to build and consume container #124

weierophinney · 2021-03-03T16:55:41Z

To speed things up further, we can build and release the actual container on release publication, and then have the action itself consume the image instead of the Dockerfile.

Additionally, we can force the release process to use the latest set of changes by defining a local GHA that uses the Dockerfile instead.
I've proven this works at https://github.com/weierophinney/test-local-action/, where I observed that the build happens relative to the checkout directory, and not to where the action is defined.

This will require defining two secrets in this repository:

CONTAINER_USERNAME (username on ghcr.io)
CONTAINER_PAT (personal authentication token with container write permissions)

Once setup, this should automatically build and push tags for the docker image, and the action will use the image to do its work.

weierophinney · 2021-03-03T16:57:53Z

I've added the secrets at this point. After first build, we'll need to make the container public, and then associate it with this repository; I'll go ahead and do that when released.

To speed things up further, we can build and release the actual container on release publication, and then have the action itself consume the image instead of the Dockerfile. Additionally, we can force the release process to use the latest set of changes by defining a local GHA that uses the Dockerfile instead. I've proven this works at https://github.com/weierophinney/test-local-action/, where I observed that the build happens relative to the checkout directory, and not to where the action is defined. Signed-off-by: Matthew Weier O'Phinney <matthew@weierophinney.net>

weierophinney · 2021-03-03T19:08:16Z

.github/workflows/build-and-push-containers.yml

+          MAJOR="${PREFIX}:$(echo ${TAG} | cut -d. -f1)"
+          MINOR="${MAJOR}.$(echo ${TAG} | cut -d. -f2)"
+          PATCH="${PREFIX}:${TAG}"
+          echo "::set-output name=tags::${MAJOR}%0A${MINOR}%0A${PATCH}"


GHA replaces %0A with \n, making these line-delimited tags. This creates a list that looks like:

ghcr.io/laminas/automatic-releases:1 ghcr.io/laminas/automatic-releases:1.11 ghcr.io/laminas/automatic-releases:1.11.0

which is then passed as the variable "tags" to the next step.

This is probably not the best idea to rely on undocumented behavior. Seems like an implementation detail that may behave differently in the future.

However, they have documented to pass as json, if you need something like this:

name: build on: push jobs: job1: runs-on: ubuntu-latest outputs: matrix: ${{ steps.set-matrix.outputs.matrix }} steps: - id: set-matrix run: echo "::set-output name=matrix::{\"include\":[{\"project\":\"foo\",\"config\":\"Debug\"},{\"project\":\"bar\",\"config\":\"Release\"}]}" job2: needs: job1 runs-on: ubuntu-latest strategy: matrix: ${{fromJSON(needs.job1.outputs.matrix)}} steps: - run: build

https://docs.github.com/en/actions/reference/context-and-expression-syntax-for-github-actions#example-returning-a-json-object

@glensc This is actually the approach recommended by the GHA team, and I discovered it here: https://git.luolix.topmunity/t/set-output-truncates-multiline-strings/16852/3 (they are indicating that they transform ANSI escape sequences emitted in outputs).

A later note in that thread indicates that it's undocumented, but that they will be updating the docs to reflect this.

I've done some thinking on this, and decided to go with the JSON approach; I'm now doing:

with: tags: ${{ join(fromJSON(steps.tags.outputs.tags), "\n") }}

(as they "tags" value is expected to be multiline).

This should address any concerns about ANSI escape sequences, and ensure we are forwards-compatible with any changes GHA makes.

Update: using "," as the separator, as CSV is supported for that value, and the "\n" value was causing issues with the GHA expressions.

weierophinney · 2021-03-03T19:09:44Z

.github/actions/automatic-releases/action.yml

+runs:
+  using: 'docker'
+  image: '../../../Dockerfile'
+  args:
+    - ${{ inputs.command-name }}


What I discovered is that while the image is in a directory relative to this one, the actual build happens relative to $GITHUB_WORKSPACE, so everything works correctly. By doing this, we get the benefits of whatever changes we've made for this release... when creating the release.

weierophinney · 2021-03-03T19:11:01Z

action.yml

@@ -15,6 +15,6 @@ inputs:

 runs:
  using: 'docker'
-  image: 'Dockerfile'
+  image: 'docker://ghcr.io/laminas/automatic-releases:1'


This is the magic: the action now uses an image on ghcr.io, which means only a pull operation is required, and no build operations!

weierophinney · 2021-03-03T20:13:42Z

.github/workflows/automatic-release.yml

@@ -17,7 +17,7 @@ jobs:
        uses: "actions/checkout@v2"

      - name: "Release"
-        uses: "./"
+        uses: "./.github/actions/automatic-releases/"


And this part ensures we use that local action, which builds from the Dockerfile directly instead of using the image.

To address concerns with using ANSI escape sequences to create multiline outputs, this patch does the following: - Emits a JSON array instead. - Creates a single job, combining steps from each previous job; this prevents the need to use `needs`, making it possible to test the solution locally. Signed-off-by: Matthew Weier O'Phinney <matthew@weierophinney.net>

GHA does not like newlines in strings used as part of the expression syntax. Since docker/build-push-action allows CSV, comma separated will accomplish the same goals. Signed-off-by: Matthew Weier O'Phinney <matthew@weierophinney.net>

bendavies · 2021-03-16T14:40:59Z

.github/workflows/build-and-push-containers.yml

+  release-container:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Compile tag list


you could use https://github.com/crazy-max/ghaction-docker-meta here

True - but this is fast and doesn't require any additional overhead as it's using tooling already in the environment.

Thanks for the link, though - I'd not seen that one!

Ocramius

Besides minor improvement suggestions, LGTM 👍

Ocramius · 2021-03-29T15:19:19Z

.github/workflows/build-and-push-containers.yml

+          PATCH="${PREFIX}:${TAG}"
+          echo "::set-output name=tags::[\"${MAJOR}\",\"${MINOR}\",\"${PATCH}\"]"
+      - name: Checkout
+        uses: actions/checkout@v2


Not super-important, but some spacing (blank lines around blocks) would go a long way with improving readability

Ocramius · 2021-03-29T15:20:15Z

.github/workflows/build-and-push-containers.yml

+        uses: docker/login-action@v1
+        with:
+          registry: ghcr.io
+          username: ${{ secrets.CONTAINER_USERNAME }}


These secrets should be prefixed, to make it clear they are local to our build

Done; used AUTOMATIC_RELEASES_ as the prefix.

Ocramius · 2021-03-29T15:21:07Z

.github/workflows/build-and-push-containers.yml

+          file: ./Dockerfile
+          platforms: linux/amd64
+          push: true
+          tags: ${{ join(fromJSON(steps.tags.outputs.tags), ",") }}


So we push each image as 3 tags? Ok for me, hard to read/follow if not versed in github actions (requires comments around it)

Yes - actual release version, minor release version, major release version.

Rationale is that if we change something in a new minor release that causes issues for somebody, they can pin to the previous minor. Alternately, they can pin to a specific "known good" release. (There are existing actions for creating tags that do the same; I chose to do it using standard *nix tooling, as it introduces zero additional overhead.)

Prefixed with `AUTOMATIC_RELEASES_` Signed-off-by: Matthew Weier O'Phinney <matthew@weierophinney.net>

Signed-off-by: Matthew Weier O'Phinney <matthew@weierophinney.net>

glensc · 2021-03-29T15:35:45Z

.github/workflows/build-and-push-containers.yml

+      - name: Compile tag list
+        id: tags
+        run: |
+          TAG=${GITHUB_REF/refs\/tags\//}


Suggested change

TAG=${GITHUB_REF/refs\/tags\//}

TAG=${GITHUB_REF#refs/tags/}

I'm sure I've already suggested this elsewhere.

Signed-off-by: Matthew Weier O'Phinney <matthew@weierophinney.net>

weierophinney added this to the 1.11.0 milestone Mar 3, 2021

weierophinney added the Enhancement label Mar 3, 2021

weierophinney force-pushed the feature/tag-and-release-container branch from 0518d61 to 98bf124 Compare March 3, 2021 18:47

weierophinney commented Mar 3, 2021

View reviewed changes

weierophinney force-pushed the feature/tag-and-release-container branch from e4641b2 to 8af0b90 Compare March 12, 2021 17:37

bendavies reviewed Mar 16, 2021

View reviewed changes

Ocramius approved these changes Mar 29, 2021

View reviewed changes

weierophinney added 2 commits March 29, 2021 10:33

qa: prefix secrets to make clear they are specific to this repo

9c34762

Prefixed with `AUTOMATIC_RELEASES_` Signed-off-by: Matthew Weier O'Phinney <matthew@weierophinney.net>

qa: whitespace between job steps in container build workflow

dc9d0b0

Signed-off-by: Matthew Weier O'Phinney <matthew@weierophinney.net>

glensc reviewed Mar 29, 2021

View reviewed changes

qa: simplify bash pattern matching when matching tag name

efeff72

Signed-off-by: Matthew Weier O'Phinney <matthew@weierophinney.net>

weierophinney merged commit 46d1840 into 1.11.x Mar 29, 2021

weierophinney deleted the feature/tag-and-release-container branch March 29, 2021 16:07

greg0ire mentioned this pull request Oct 3, 2021

How do I downgrade to a previous version? #167

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Modify action to build and consume container #124

Modify action to build and consume container #124

weierophinney commented Mar 3, 2021

weierophinney commented Mar 3, 2021

weierophinney Mar 3, 2021

glensc Mar 3, 2021

weierophinney Mar 4, 2021 •

edited

Loading

weierophinney Mar 12, 2021

weierophinney Mar 16, 2021

weierophinney Mar 3, 2021 •

edited

Loading

weierophinney Mar 3, 2021

weierophinney Mar 3, 2021

bendavies Mar 16, 2021

weierophinney Mar 16, 2021

Ocramius left a comment

Ocramius Mar 29, 2021

weierophinney Mar 29, 2021

Ocramius Mar 29, 2021

weierophinney Mar 29, 2021

Ocramius Mar 29, 2021

weierophinney Mar 29, 2021

glensc Mar 29, 2021

weierophinney Mar 29, 2021

Modify action to build and consume container #124

Modify action to build and consume container #124

Conversation

weierophinney commented Mar 3, 2021

weierophinney commented Mar 3, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

weierophinney Mar 4, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

weierophinney Mar 3, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ocramius left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

weierophinney Mar 4, 2021 •

edited

Loading

weierophinney Mar 3, 2021 •

edited

Loading