Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build(ci): build docker images for each tag, release, push #151

Merged
merged 7 commits into from
Sep 2, 2024

Conversation

psyray
Copy link
Contributor

@psyray psyray commented Aug 20, 2024

Fix #106

First PoC to automate docker images creation
This need tests

Here are the main modifications and their explanations:

  1. I added triggers for pull requests, pushes to master and release/**, tags v*.*.*, and published releases.

  2. I defined environment variables for the registry, owner, and project name to facilitate maintenance.

  3. I used a matrix to build all images defined in tasks.json.

  4. I modified the docker/metadata-action configuration to generate appropriate tags, including the latest tag for the default branch.

  5. I adjusted the context and Dockerfile path in docker/build-push-action to match your project structure.

  6. I added an update-release job that runs only when a release is published. This job updates the release description with the list of Docker images built.

This configuration will automatically build and push all images to GHCR for each pull request (without pushing), each push to master and release/**, each v*.*.* tag, and each published release. The images will be tagged with the version number, commit SHA, and latest for the default branch.

Remember to grant the necessary permissions to the GitHub action to push to GHCR and update releases. You can do this in your GitHub repository settings.

@psyray psyray self-assigned this Aug 20, 2024
@psyray psyray added the enhancement New feature or request label Aug 20, 2024
@psyray psyray modified the milestone: v2.1.0 release Aug 20, 2024
@psyray psyray linked an issue Aug 20, 2024 that may be closed by this pull request
3 tasks
@psyray
Copy link
Contributor Author

psyray commented Aug 20, 2024

Next steps

  1. Create a bot account for the organization:

    • Create a new GitHub account that will serve as a bot for your organization.
    • Add this bot account to your organization with the necessary permissions.
  2. Create a Personal Access Token (PAT) for this bot account:

    • Log in with the bot account.
    • Generate a PAT with the necessary permissions (minimum read:packages, write:packages, et delete:packages).
  3. Add the PAT as a secret at the organization level:

    • Go to the organization settings.
    • Navigate to "Secrets and variables" > "Actions".
    • Create a new organization secret (e.g., ORG_GHCR_PAT) and add the PAT as the value.
  4. Modify build.yml file to use this secret and the bot name:

env:
  ...
  BOT_NAME: your-org-bot-name

jobs:
  build-and-push:
    runs-on: ubuntu-latest
    steps:
      # ... other steps ...

      - name: Login to GHCR
        uses: docker/login-action@v3
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ env.BOT_NAME }}
          password: ${{ secrets.ORG_GHCR_PAT }}
...

In this configuration:

  • BOT_NAME is defined as an environment variable with your bot account name.
  • The ORG_GHCR_PAT secret is used for authentication, which is an organization-level secret.

This approach has several advantages:

  1. All actions are performed on behalf of the organization rather than a specific individual.
  2. Permissions can be managed more easily at the organization level.
  3. If a maintainer leaves the organization, there's no need to update the workflows.

Remember to give the bot account the necessary permissions in your organization and in the GitHub repository settings so it can push images to GHCR and perform other necessary actions.

@psyray psyray marked this pull request as ready for review August 25, 2024 17:37
@AnonymousWP
Copy link
Member

Please look into the failing CI. I just tried to re-run it (with debug logs enabled), but no luck. See here for logs: https://github.com/Security-Tools-Alliance/rengine-ng/actions/runs/10548890515/job/29244740288?pr=151#step:7:4435

#10 18.53 Package firefox is not available, but is referred to by another package.
#10 18.53 This may mean that the package is missing, has been obsoleted, or
#10 18.53 is only available from another source
#10 18.53 However the following packages replace it:
#10 18.53   gnome-browser-connector
#10 18.53 
#10 18.53 E: Package 'firefox' has no installation candidate
#10 ERROR: process "/bin/sh -c wget -q https://packages.mozilla.org/apt/repo-signing-key.gpg -O- | tee /etc/apt/keyrings/packages.mozilla.org.asc &&     gpg -n -q --import --import-options import-show /etc/apt/keyrings/packages.mozilla.org.asc | awk '/pub/{getline; gsub(/^ +| +$/,\"\"); print \"\\n\"$0\"\\n\"}' &&     echo \"deb [signed-by=/etc/apt/keyrings/packages.mozilla.org.asc] https://packages.mozilla.org/apt mozilla main\" | tee -a /etc/apt/sources.list.d/mozilla.list &&     echo 'Package: *Pin: origin packages.mozilla.orgPin-Priority: 1000' | tee /etc/apt/preferences.d/mozilla && apt update -y && apt install -y firefox" did not complete successfully: exit code: 100
------
 > [ 5/17] RUN wget -q https://packages.mozilla.org/apt/repo-signing-key.gpg -O- | tee /etc/apt/keyrings/packages.mozilla.org.asc &&     gpg -n -q --import --import-options import-show /etc/apt/keyrings/packages.mozilla.org.asc | awk '/pub/{getline; gsub(/^ +| +$/,""); print "\n"$0"\n"}' &&     echo "deb [signed-by=/etc/apt/keyrings/packages.mozilla.org.asc] https://packages.mozilla.org/apt mozilla main" | tee -a /etc/apt/sources.list.d/mozilla.list &&     echo 'Package: *Pin: origin packages.mozilla.orgPin-Priority: 1000' | tee /etc/apt/preferences.d/mozilla && apt update -y && apt install -y firefox:


18.51 Reading state information...
18.53 Package firefox is not available, but is referred to by another package.
18.53 This may mean that the package is missing, has been obsoleted, or
18.53 is only available from another source
18.53 However the following packages replace it:
18.53   gnome-browser-connector
18.53 
18.53 E: Package 'firefox' has no installation candidate
------
Dockerfile:48
--------------------
  47 |     
  48 | >>> RUN wget -q https://packages.mozilla.org/apt/repo-signing-key.gpg -O- | tee /etc/apt/keyrings/packages.mozilla.org.asc && \
  49 | >>>     gpg -n -q --import --import-options import-show /etc/apt/keyrings/packages.mozilla.org.asc | awk '/pub/{getline; gsub(/^ +| +$/,""); print "\n"$0"\n"}' && \
  50 | >>>     echo "deb [signed-by=/etc/apt/keyrings/packages.mozilla.org.asc] https://packages.mozilla.org/apt mozilla main" | tee -a /etc/apt/sources.list.d/mozilla.list && \
  51 | >>>     echo '\
  52 | >>> Package: *\
  53 | >>> Pin: origin packages.mozilla.org\
  54 | >>> Pin-Priority: 1000\
  55 | >>> ' | tee /etc/apt/preferences.d/mozilla && apt update -y && apt install -y firefox
  56 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c wget -q https://packages.mozilla.org/apt/repo-signing-key.gpg -O- | tee /etc/apt/keyrings/packages.mozilla.org.asc &&     gpg -n -q --import --import-options import-show /etc/apt/keyrings/packages.mozilla.org.asc | awk '/pub/{getline; gsub(/^ +| +$/,\"\"); print \"\\n\"$0\"\\n\"}' &&     echo \"deb [signed-by=/etc/apt/keyrings/packages.mozilla.org.asc] https://packages.mozilla.org/apt mozilla main\" | tee -a /etc/apt/sources.list.d/mozilla.list &&     echo 'Package: *Pin: origin packages.mozilla.orgPin-Priority: 1000' | tee /etc/apt/preferences.d/mozilla && apt update -y && apt install -y firefox" did not complete successfully: exit code: 100
::group::Reference
Reference
::group::Check build summary support
Check build summary support
Error: buildx failed with: ERROR: failed to solve: process "/bin/sh -c wget -q https://packages.mozilla.org/apt/repo-signing-key.gpg -O- | tee /etc/apt/keyrings/packages.mozilla.org.asc &&     gpg -n -q --import --import-options import-show /etc/apt/keyrings/packages.mozilla.org.asc | awk '/pub/{getline; gsub(/^ +| +$/,\"\"); print \"\\n\"$0\"\\n\"}' &&     echo \"deb [signed-by=/etc/apt/keyrings/packages.mozilla.org.asc] https://packages.mozilla.org/apt mozilla main\" | tee -a /etc/apt/sources.list.d/mozilla.list &&     echo 'Package: *Pin: origin packages.mozilla.orgPin-Priority: 1000' | tee /etc/apt/preferences.d/mozilla && apt update -y && apt install -y firefox" did not complete successfully: exit code: 100

@psyray
Copy link
Contributor Author

psyray commented Aug 26, 2024

Please look into the failing CI. I just tried to re-run it (with debug logs enabled), but no luck. See here for logs: https://github.com/Security-Tools-Alliance/rengine-ng/actions/runs/10548890515/job/29244740288?pr=151#step:7:4435

It should be ok now.
This PR need rebase to have the latest Dockerfile from release/2.1.0

@AkechiShiro
Copy link

AkechiShiro commented Aug 31, 2024

I didn't find any image on the docker hub, is this only built in CI/CD but not yet published on Docker Hub or elsewhere ? Was this PR rebased ?

@AnonymousWP
Copy link
Member

I didn't find any image on the docker hub, is this only built in CI/CD but not yet published on Docker Hub or elsewhere ? Was this PR rebased ?

This will only work once it's merged into 2.1.0, and then another PR gets merged to 2.1.0. That should be the case, but we can only test it by merging. I'm reviewing the code now.

@AnonymousWP
Copy link
Member

This configuration will automatically build and push all images to GHCR for each pull request (without pushing)

I don't understand this. First you're saying it automatically builds and pushes all images, but then later you say "without pushing".

Create a bot account for the organization:

I'm not a fan of this, because we'll have to maintain the security credentials of that account too:

  • Who's going to be responsible for that account within our org?
  • Who will have its credentials?
  • Where will we store it?
  • What are we going to do to prevent credential leaking?

@AnonymousWP AnonymousWP added ci This is for CI-related PRs and removed enhancement New feature or request labels Sep 2, 2024
@psyray
Copy link
Contributor Author

psyray commented Sep 2, 2024

This configuration will automatically build and push all images to GHCR for each pull request (without pushing)

I don't understand this. First you're saying it automatically builds and pushes all images, but then later you say "without pushing".

You're right I was certainly tired, I will change this

.vscode/tasks.json Outdated Show resolved Hide resolved
.vscode/tasks.json Outdated Show resolved Hide resolved
@psyray
Copy link
Contributor Author

psyray commented Sep 2, 2024

I didn't find any image on the docker hub, is this only built in CI/CD but not yet published on Docker Hub or elsewhere ? Was this PR rebased ?

Images are not on docker hub but in GitHub Container Registry (GHCR)
As @AnonymousWP said images will be available only with 2.1.0 release
You could display images in the packages tab of the repository

@AnonymousWP AnonymousWP merged commit 5ca5915 into release/2.1.0 Sep 2, 2024
15 checks passed
@AnonymousWP AnonymousWP deleted the 106-ci-automate-docker-images-creation branch September 2, 2024 16:37
@AkechiShiro
Copy link

Thanks @psyray as well as @AnonymousWP for the merge and the work on this project !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci This is for CI-related PRs
Projects
None yet
Development

Successfully merging this pull request may close these issues.

ci(docker): automate creation of docker image to speedup install/start
3 participants