Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add GitHub workflow file to build/tag/push custom docker images #6525

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

Exairnous
Copy link
Contributor

This allows docker images with a custom tag/name to be created on demand for any branch containing this workflow and pushed to the specified docker repository. Two versions of the docker image will always be pushed, one ending with -latest and one ending with the workflow run number, e.g. my-custom-image-name-latest and my-custom-image-name-42.

This should help make it easier to customize Community Edition instances, test pull requests/development branches, and to provide docker images of long running branches (large alpha/beta features) for community testing.

Once this is finalized, it will be added to the other repositories that can benefit from it as well.

@Exairnous Exairnous force-pushed the add-custom-docker-build-github-workflow branch from d7ba381 to db33099 Compare December 4, 2024 08:58
@Exairnous
Copy link
Contributor Author

I'm not sure where the documentation for this workflow should live (the wiki for each repository, the hubs-docs repo, somewhere else?), so for now I'm just going to post it here in this comment:

Building Custom Docker Images

In each repository there is a GitHub Workflow/Action named custom-docker-build-push for building custom docker images and pushing them to a docker/container registry of your choice. Simply run this Action on any branch, with the appropriate details filled in (either through the predefined defaults or by the overrides), and you will end up with a docker image of that branch that can be used in any Community Edition instance.

The Action allows you almost complete control over where your docker images are sent and how they are tagged. The details required for this to work can either be provided at runtime or by preset defaults that you set up.

Options/Setup

The following properties can be configured/overridden. Any values that are manually specified for that run will always be used. Any fields not passed values will fall back on predefined variables/secrets (if they have been created).

Options:

  • The branch the workflow will run on. [REQUIRED]
    • This has to be manually specified each time when running the Action.
  • The registry base URL. [REQUIRED]
    • This is used to specify which docker/container registry to push to.
    • Common values include: docker.io for Docker Hub, and ghcr.io for GitHub. For other registries, check their documentation on what to put here.
    • Create a repository variable named REGISTRY_BASE_URL to set a default registry to use.
  • The registry username. [REQUIRED]
    • The username of the account for the registry you want to push to.
    • Create a repository secret named REGISTRY_USERNAME to set a default username to use.
  • The registry password. [REQUIRED]
    • The password of the account for the registry you want to push to.
    • Create a repository secret named REGISTRY_PASSWORD to set a default password to use.
  • The registry namespace. [REQUIRED]
    • This is often the same as the username, but some registries allow you to manually specify a namespace.
    • Create a repository variable named REGISTRY_NAMESPACE to set a default namespace to use.
  • The image tag. [OPTIONAL]
    • This is basically the name of your image.
    • The default value of this is the branch name.
    • When the workflow is run, two images/tags will be pushed to the registry, one will be the <image_tag_value>-latest and the other will be the <image_tag_value>-<action_run_number>, e.g. my-cool-branch-latest and my-cool-branch-42
  • The name of the Dockerfile [OPTIONAL]
    • This has been preset and you shouldn't need to change it unless you've changed the name/location of the Dockerfile in your code. If you have changed the name/location of your Dockerfile you will need to specify this each and every time you run the Action or update the default in the GitHub workflow file.
  • The code path. [OPTIONAL]
    • The path to the code you want built. Used for only building a specific part of the repo.
    • This doesn't usually need to be filled/changed.
  • The build cache.
    • Using the build cache will speed up subsequent builds of your images, but can sometimes cause problems with registries.
    • This is enabled by default, but if you run into problems building, you can disable it. The build will be slower, but it should hopefully work.
    • Example error to watch out for: buildx failed with: ERROR: failed to solve: error writing manifest blob: failed commit on ref "sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx": unexpected status from PUT request to https://xx.xxxxxxx.xxx/xx/namespace/repository/manifests/buildcache: 400

This allows docker images with a custom tag/name to be created on demand for any branch containing this workflow and pushed to the specified docker repository.  Two versions of the docker image will always be pushed, one ending with -latest and one ending with the workflow run number, e.g. my-custom-image-name-latest and my-custom-image-name-42.

The details required for the workflow to run can either be provided by predefined repository secrets/variables or specified manually when invoking the workflow.

This should help make it easier to customize Community Edition instances, test pull requests/development branches, and to provide docker images of long running branches (large alpha/beta features) for community testing.
@Exairnous Exairnous force-pushed the add-custom-docker-build-github-workflow branch from db33099 to da8af27 Compare December 4, 2024 09:19
@Exairnous
Copy link
Contributor Author

@mikemorran I've credited Brandon Patterson as mentioned here: https://github.com/Hubs-Foundation/hubs-blender-files/blob/main/creator-labs-files/CL-archive/ghost-to-md-output/2023-10-16-community-edition-case-study-quick-start-on-gcp-w-aws-services.md, but should I be crediting you as well? Also, I see there are two steps for the docker/setup-buildx-action, do you know if there is a reason for this? As far as I can tell, having duplicate build steps doesn't cause any problems, but I'd like to either remove one or add a comment explaining why there are two.

Exairnous added a commit to Hubs-Foundation/hubs-docs that referenced this pull request Dec 18, 2024
Adds a new page in the developer section for GitHub workflows with documentation for the Custom Docker Build Push GitHub workflow.
This provides docs for the GitHub workflow introduced here:
Hubs-Foundation/hubs#6525
@Exairnous Exairnous marked this pull request as ready for review December 18, 2024 12:47
@Exairnous
Copy link
Contributor Author

@DougReeder @hrithikwins I've added a docs PR for this to the hubs-docs repository: Hubs-Foundation/hubs-docs#198
If everything looks alright to you with these PRs, then I'll probably merge them soon. I will update the wiki once everything is done to point to the hubs-docs page on workflows.

While it would be nice to get the answers to the questions I asked @mikemorran I don't think they affect the functionality so we can update stuff later, if needed.

@mikemorran
Copy link
Contributor

@Exairnous I originally wrote all of the tutorial steps and @zfox23 wrote the introductory stuff. @pattersonbl2 was an advisor and great help on all of the information, but I am not aware that he wrote anything in the document.

I'm trying to see what you are referencing with "two steps for docker/setup-buildx-action. Can you clarify what you are asking about on this?

@Exairnous
Copy link
Contributor Author

@mikemorran In your article/tutorial you wrote: "When you have made your customizations and are ready to deploy, go to .github/workflows/ and add a file called ce-build.yml and populate with this code created by Hubs Team member, Brandon Patterson", so that sounds to me like Brandon Patterson wrote ce-build.yml (which this PR is based off of). So that's why I'm crediting him, but I was wondering if you (or now that you mention it, anyone else) contributed to that file and deserve credit.

I'm trying to see what you are referencing with "two steps for docker/setup-buildx-action. Can you clarify what you are asking about on this?

Here and here seem to be duplicates/perform the same job. I have kept both those uses: docker/setup-buildx-action sections in the workflow in this PR, but I'm not sure why the setup needs to be performed twice. My best guess is that the second one needs to be performed if the code path for multi repo is specified, but then why not just have the last one and forget about the first one? If possible, I'd like to put a comment there explaining this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants