Build Docker image and push to GHCR #230

br3ndonland · 2024-04-19T01:51:01Z

Description

Closes #58

Up to this point, the project has been set up as a Docker action referencing the Dockerfile.

Lines 86 to 88 in 3fbcf7c

    
           runs: 
        
             using: docker 
        
             image: Dockerfile

The downside to using the Dockerfile for the action is that the Docker image must be built every time the action is used (#58).

This PR will set up the project to build the Docker image and push it to GitHub Container Registry (GHCR). This change will speed up user workflows every time the action is used because the workflows will simply pull the Docker image from GHCR instead of building again.

Changes

Build container image with GitHub Actions

This PR will build Docker images with the Docker CLI (docker build). Builds will include inline cache metadata so layers can be reused by future builds.

This PR only proposes to build container images for x86_64 (linux/amd64) because GitHub Actions Linux runners currently only support x86_64 CPU architectures (actions/runner-images#5631), and this project only supports GitHub Actions Linux runners. The README explains:

Since this GitHub Action is docker-based, it can only be used from within GNU/Linux based jobs in GitHub Actions CI/CD workflows. This is by design and is unlikely to change due to a number of considerations we rely on.

Push container image to GHCR

The workflow will log in to GHCR using the built-in GitHub token and push the Docker image. Workflow runs triggered by pull requests will build the Docker image and run the smoke tests but will not push the Docker image.

Update action to pull container image from GHCR

Docker actions support pulling in pre-built Docker images by supplying a registry address to the image: key. The downside to this syntax is that there's no way to specify the correct Docker tag because the GitHub Actions image: and uses: keys don't accept any context. For example, if a user's workflow has uses: pypa/gh-action-pypi-publish@release/v1.8, then the action should pull in a Docker image built from the release/v1.8 ref, something like ghcr.io/pypa/gh-action-pypi-publish:release-v1.8 (Docker tags can't have /).

# this works but the image tag can't be customized
runs:
  using: docker
  image: docker://ghcr.io/pypa/gh-action-pypi-publish:release-v1.8

# this doesn't work because `image:` doesn't support context
runs:
  using: docker
  image: docker://ghcr.io/pypa/gh-action-pypi-publish:${{ github.action_ref }}

The workaround is to switch the top-level action.yml to a composite action that then calls the Docker action, substituting the correct image name and tag.

Originally I thought that I'd have a workflow where I trigger a release, that release adds a commit that hardcodes an update to action.yml with the "future" version tag, tags that commit and pushes it (post docker publish). It wouldn't be on the main branch, the tags would be on the orphaned leaves.

This looks like a better idea so far. Thanks again!

pypa#230 (comment)

br3ndonland · 2024-04-26T23:12:16Z

That sounds great. Take your time. Thanks for your consideration.

If you do decide to accept this change, I'm happy to help maintain the workflows in the future. Feel free to mention me @br3ndonland and I will help address any issues that come up.

pypa#230 (comment)

webknjaz · 2024-05-16T15:56:33Z

Thanks! I've hit "rebase" on the UI to get this on top of the recent changes/linting/lockfile bumps but haven't yet looked into it deeper.

.github/workflows/build-and-push-docker-image.yml

.github/actions/run-docker-container/action.yml

.github/workflows/build-and-push-docker-image.yml

webknjaz · 2024-05-16T19:18:18Z

.github/workflows/self-smoke-test-action.yml

  pull_request:
+  workflow_run:


Two separate workflow runs is often hard to track. Instead, I adopted a practice of modularizing the workflow pieces as reusable workflows having the reusable- prefix in their names. This allows embedding everything in all the right places. Let's try this, WDYT?

I will return to this suggestion at a later time.

I think we should be able to circle back to this already...

Two separate workflow runs is often hard to track. Instead, I adopted a practice of modularizing the workflow pieces as reusable workflows having the reusable- prefix in their names. This allows embedding everything in all the right places. Let's try this, WDYT?

@webknjaz I removed workflow_call and switched to a reusable workflow instead.

Note that the required status checks for the repo will need to be updated.

.github/workflows/self-smoke-test-action.yml

.pre-commit-config.yaml

Dockerfile

webknjaz · 2024-05-16T19:37:27Z

action.yml

+  steps:
+  - name: Reset path if needed
+    run: |
+      # Reset path if needed


Why would this be needed outside the container?

Because you've set up a test that modifies the $PATH (#112, 1350b8b).

gh-action-pypi-publish/.github/workflows/self-smoke-test-action.yml

Lines 85 to 90 in 699cd61

- name: ✅ Smoke-test the locally checked out action

uses: ./test

env:

DEBUG: >-

true

PATH: utter-nonsense

action.yml

.github/actions/run-docker-container/action.yml

action.yml

webknjaz · 2024-05-16T20:01:27Z

I'm done with the initial review. More is needed, but I'd rather accept what I can through separate PRs to make this one smaller. And the suggested refactoring could be done in parallel. I think that generating the file is a good idea. It should be possible to write the file without bringing in the PyYAML dependency. But it's not that easy for reading it. Can we make use of yq somehow, and convert YAML to JSON this way, maybe?

br3ndonland · 2024-05-27T20:43:49Z

@webknjaz thank you for your detailed review. I've addressed most of your comments so far.

webknjaz · 2024-05-29T14:40:43Z

Commits like 213c885ac41d769527ac150e2e633bb1ccd886d4 aren't really necessary since Git would automatically absorb changes applied separately. FYI. In fact, this one may have a harmful effect — when the label change is merged into the default branch, and this one rebased on top, Git will keep the removal commit and attempt deleting the label 🤷‍♂️

Anyway, we'll address this later on.

create-docker-action.py

action.yml

requirements/github-actions.txt

action.yml

create-docker-action.py

webknjaz · 2024-05-29T20:03:55Z

@br3ndonland I'm sure my review is incomplete, but hopefully it gives you enough ideas to try out until the next time.

I think it'd be nice to get this in before #236 if at all possible.

webknjaz · 2024-05-29T20:28:13Z

@br3ndonland could you also rebase this branch locally? The GH button doesn't work, meaning there's going to be some conflicts to resolve.

webknjaz · 2024-06-16T20:05:14Z

action.yml

+    if: runner.os != 'Linux'
+    run: |
+      >&2 echo This action is only able to run under GNU/Linux environments
+      exit 1


Could you add CI jobs testing that this fails under corresponding envs?

I added CI jobs that test failure on macOS and Windows.

webknjaz · 2024-06-16T20:08:32Z

.github/workflows/build-and-push-docker-image.yml

+      tag:
+        description: Docker image tag


Can we incorporate the prior art from the discussions in #45 and accept the 3-segment version as an input, then deduce everything else from that + push the Git tag and advance the proper branches? Perhaps, this could also happen before this PR so it exists as a separate base.

Can we incorporate the prior art from the discussions in #45 and accept the 3-segment version as an input, then deduce everything else from that + push the Git tag and advance the proper branches? Perhaps, this could also happen before this PR so it exists as a separate base.

I don't understand what you're asking here. You want separate inputs for major, minor, and patch versions? What if you're building the Docker image from the release/v1 branch? What is the "3-segment version" in that case?

@br3ndonland I was talking about the workflow inputs. Everything else should be computed from them.

The current workflow for me is as follows:

Tag v1.10.1 on top of unstable/v1

Fast-forward release/v1 to v1.10.1

Fast-forward release/v1.10 to v1.10.1 (or create it)

git push --atomic everything in one go

Create a GitHub Release from the UI

With this, my 3-segment “input” is 1.10.1 and v1+v1.10 are being extracted from that. Do you think we could bake containers into this flow with not many changes? This will involve tagging the same image with multiple tags, as I understand. And the end-users referencing versions by both tags and branches should be able to retrieve whatever container there is, right?

@webknjaz thanks for explaining that. I think we're actually pretty close to the desired workflow. There's just a small update needed to the Docker build workflow so it pushes tags with the major and minor version numbers. I've pushed that change.

Your updated workflow would look like this:

Tag v1.10.1 on top of unstable/v1

Fast-forward release/v1 to v1.10.1

Fast-forward release/v1.10 to v1.10.1 (or create it)

git push --atomic everything in one go

Trigger the 🏗️ workflow from the Git tag. The workflow will push the Docker image with three Docker tags:

ghcr.io/pypa/gh-action-pypi-publish:v1.10.1

ghcr.io/pypa/gh-action-pypi-publish:v1.10

ghcr.io/pypa/gh-action-pypi-publish:v1

Create a GitHub Release from the UI

pypa#230 (comment)

pypa#230 (comment) Co-authored-by: Sviatoslav Sydorenko (Святослав Сидоренко) <wk.cvs.github@sydorenko.org.ua>

This project uses pypa/gh-action-pypi-publish to publish Python packages to PyPI with an OIDC trusted publisher (6e532c6). pypa/gh-action-pypi-publish is set up as a Docker action referencing the Dockerfile. The downside to using the Dockerfile for the action is that the Docker image must be built every time the action is used. This will hopefully change in the near future if Docker images are pre-built and pushed to a registry (pypa/gh-action-pypi-publish#230). In the meantime, this commit will move related steps to a dedicated GitHub Actions job so that the Docker image is not built every time GitHub Actions jobs run. 6e532c6 https://docs.github.com/en/actions/creating-actions/creating-a-docker-container-action https://docs.pypi.org/trusted-publishers/ https://github.com/pypa/gh-action-pypi-publish

This project uses pypa/gh-action-pypi-publish to publish Python packages to PyPI with an OIDC trusted publisher (59ec546). pypa/gh-action-pypi-publish is set up as a Docker action referencing the Dockerfile. The downside to using the Dockerfile for the action is that the Docker image must be built every time the action is used. This will hopefully change in the near future if Docker images are pre-built and pushed to a registry (pypa/gh-action-pypi-publish#230). In the meantime, this commit will move related steps to a dedicated GitHub Actions job so that the Docker image is not built every time GitHub Actions jobs run. 59ec546 https://docs.github.com/en/actions/creating-actions/creating-a-docker-container-action https://docs.pypi.org/trusted-publishers/ https://github.com/pypa/gh-action-pypi-publish

webknjaz · 2024-09-04T13:13:55Z

@br3ndonland this also needs to be rebased now that the attestations PR is in.

Up to this point, the project has been set up as a Docker action referencing the Dockerfile. The downside to using the Dockerfile for the action is that the Docker image must be built every time the action is used. This commit will set up the project to build the Docker image and push it to GitHub Container Registry (GHCR). This change will speed up user workflows every time the action is used because the workflows will simply pull the Docker image from GHCR instead of building again. Changes: - Add required metadata to Dockerfile - Build container image with GitHub Actions - Push container image to GHCR Docker actions support pulling in pre-built Docker images. The downside is that there's no way to specify the correct Docker tag because the GitHub Actions `image` and `uses:` keys don't accept any context. For example, if a user's workflow has `uses: pypa/gh-action-pypi-publish@release/v1.8`, then the action should pull in a Docker image built from the `release/v1.8` branch, something like `ghcr.io/pypa/gh-action-pypi-publish:release-v1.8` (Docker tags can't have `/`). The workaround is to switch the top-level `action.yml` to a composite action that then calls the Docker action, substituting the correct image name and tag.

pypa#230 (comment)

actions/checkout#27 (comment) actions/checkout#1108

pypa#230 (comment) Co-authored-by: Sviatoslav Sydorenko (Святослав Сидоренко) <wk.cvs.github@sydorenko.org.ua>

Co-authored-by: Sviatoslav Sydorenko (Святослав Сидоренко) <wk.cvs.github@sydorenko.org.ua>

for more information, see https://pre-commit.ci

webknjaz reviewed Apr 25, 2024

View reviewed changes

.github/workflows/build-and-push-docker-image.yml Show resolved Hide resolved

webknjaz mentioned this pull request Apr 26, 2024

[Roadmap] Implement PEP 740 pypi/warehouse#15871

Open

23 tasks

br3ndonland added a commit to br3ndonland/gh-action-pypi-publish that referenced this pull request Apr 26, 2024

Separate docker login and docker push

bb0315f

pypa#230 (comment)

webknjaz pushed a commit to br3ndonland/gh-action-pypi-publish that referenced this pull request May 16, 2024

Separate docker login and docker push

3384fad

pypa#230 (comment)

webknjaz force-pushed the ghcr branch from bb0315f to 3384fad Compare May 16, 2024 15:55