Automating releases, building and publication of package artefacts

[almereyda]

Hi! I have observed the Federated Wiki build toolchain is "missing" automated builds to NPM and the GitHub package repository or an OCI registry.

The current release processes depend on manual work by @paul90 in regular exchange with @WardCunningham. In times of Microsoft pleasing the world with GitHub Actions, we can choose to accept their offering and have it help us to automate releases "away", by reducing a few manual steps.

Upstream

This can benefit by a generalized workflow repository in the fedwiki organisation, from which manifests may be reused. It could carry this forge's brand in the .github repository, be an actions repository or carry any self-chosen name, maybe just here in wiki with a little overhead in a .github branded directory. Would there be opinions on this?

When there are no objections to the overall procedure, I would like to proceed with:

• wiki-client releases trigger package build and publication for NPM and OCI
• A new wiki-client release triggers the creation of a PR that offers to increase this dependency's version here.

---

• wiki-server releases trigger package build and publication for NPM and OCI
• A new wiki-client release triggers the creation of a PR that offers to increase this dependency's version here.

Downstream

Release automation for wiki is in place that

• builds and pushes the NPM package and multiarch OCI images upon release

Later we could reuse these preparations to have the plugin repositories behave in a similar way. Also self-updating draft releases, which aggregate recently merged PRs since last tag, can save a few clicks when doing bulk operations. In tying Wiki closer to GitHub, in return we receive some support to ease maintenance chores.

Paul and Ward, would the above seem useful to you?

[almereyda]

Initial work has been done to showcase automated releases and image builds for federated wiki using the GitHub infrastructure.

After the manifests of the @federated-wiki initiative were not updated since beginning of the pandemic, they were in a good spot to exemplify the suggested procedure. Were, because they now support (semi-)automatic builds.

A federated-wiki/workflows repository holds the central build-and-push.yml manifest, which is being reused in this example, which pushes only to GitHub Packages, due to ease of authentication, since no additional secrets are involved. An example for logging in and pushing to Docker Hub is contained in the comments of the file.

The federated-wiki/wiki-docker repository has been modified to run without a privileged user inside the container, and offers multiple ways to trigger a build:

Using the detect and tag new version (manifest) workflow

• By merging a commit to main that increases the WIKI_TAG in the Dockerfile
  In case the trigger evaluates positively, a tagged release is generated and a build by the CI workflow below is started
• increase wiki version to 0.31.0 federated-wiki/wiki-oci-base#4
• federated-wiki/wiki-docker#4 workflow run

Using the CI workflow (manifest) directly allows to create interim releases, which are additional to the upstream versions.

• By creating a new tagged release on the GitHub releases page or via API
• By using the workflow_dispatch event trigger
  

This way it is easy to automatically track upstream wiki versions, and to offer pull requests that increase the package version, while allowing to create manual interim releases for different purposes, mainly patch and test releases.

The federated-wiki/wiki-docker-plugins repository, which is used for assembling a wiki distribution, adopts a simpler CI workflow (manifest) than the previous version, and basically only contains triggers and build configuration. The Containerfile has as well been adapted to work without root inside the container.

The artefacts are uploaded to the Packages repository of the GitHub organisation, here federated wiki.

• ghcr.io/federated-wiki/base
• ghcr.io/federated-wiki/distribution

Due to suggested snippets in the Docker, Node.js and GitHub documentation, the build is performed for both the amd64 and arm64 (aarch64) architectures, and packages for both are made available via the same image names and tags.

Now it will be much easier and much more deterministic to create new versions of our federated wiki distribution for the local use cases, mainly the federated.wiki and commoning.wiki farms.

[paul90]

I see little benefit in adding a script that will be doing little more than simply automate npm publish. That is just adding something extra that is likely to go wrong, and would need maintaining.

I think for your upstream workflows, you are asking us to increment the wiki version when changes to any of the included wiki parts is updated. I don't see a problem with remembering to do that.

Not sure how your automation copes when we package either wiki or any of its parts with a dist-tag next, so that major changes can be tested. Hopefully the automation will simply ignore anything other than the latest tag.

[almereyda]

Thanks for looking at this. Indeed my automation only picks up from what is published to these repositories, basically the Dockerfiles there, and has no integration whatsoever with the wiki upstream here or on NPM, yet.

Marking releases as tags and pushing them here, eventually also using the GitHub releases feature on them, will only make it easier for third-parties to follow what's going on in Wiki land, and is not needed for the procedures above.

I'd still consider NPM being the canonical source for Wiki releases, which could be polled by a daily job for changes, avoiding any tight integration, e.g. with Webhooks.

Building always up-to-date images from next would be another question.

What's left then is to increase wiki versions, when one of its downstream packages change their version, too, yes. The automation could pick this up with regular polling, as mentioned above.

[almereyda]

I have now set up a clunky extension of the previous example, which will query NPM once a day for new wiki versions, in comparison to the one available there, and build a new container from it, when necessary:

• Pull and compare last NPM release · Workflow runs · federated-wiki/wiki-docker

This has been exemplified in this task run:

• Pull and compare last NPM release · federated-wiki/wiki-docker@ab0e708

It remains left to increase the base image version of the dependant distribution, once a new base image has been released.

Now I also understand your remark regarding a mutable dist version tag on NPM better.

The script queries NPM ¹ with:

curl -sL https://registry.npmjs.org/wiki/latest | jq -r .version

Whenever this is different to the previous version in the Dockerfile ², it is used for the next release:

${{ steps.dockerfile.outputs.version != steps.npm.outputs.version }}

There appear to be many ways in which this can break. For getting this right, I had to revert version increase commits multiple times, because the triggers were not working as expected. But the one that checks for new versions, which would also try to generate a new release for a downgraded version, but fail sufficiently due to the preexisting tag:

• Merge pull request #27 from federated-wiki/revert-25-0.32.1 · federated-wiki/wiki-docker@ab0e708

Let's see if that schedule actually works, and if it manages to react to new releases on its own now.

[almereyda]

There are now automated releases for the packaged distribution of wiki. I can go to

• Increase base version · Workflow runs · federated-wiki/wiki-docker-plugins

enter the old and the new version number, and will end up with new tagged releases for the distribution, both for the git tag and the OCI image tag.

• Releases · federated-wiki/wiki-docker-plugins
• Package distribution

There is still room for improvement. For example building from the implicit latest tag before did not provide much customisability, but would make sure we are always up to date when cutting a release artifact.

There are other ways to update container images when their base images changes, but for now I like the explicitness of syncing the version tag to the wiki upstream of the distribution.

Another next step will be to reactivate pushing to Docker Hub, which is more convenient to use for most.

[almereyda]

I liked what you said in #143 (comment)

Deployment specific stuff, be it docker, nix, or something else, really don't belong in the repo for the npm package. Much better that it is elsewhere in a repo with those who use it, and can maintain it.

Which totally makes sense, given that packaging exists and works.

[almereyda]

Just confirming that this works now as intended.

The last release for

• idProvider: undefined wiki-security-passportjs#41 (comment)

appeared in the morning at

• https://github.com/federated-wiki/wiki-oci-base/releases/tag/v0.33.0

which allowed me to manually initiate the upgrade with

• https://github.com/federated-wiki/wiki-oci-distribution/actions/runs/6983413609

and a new container was born:

• https://github.com/federated-wiki/wiki-oci-distribution/pkgs/container/distribution

Now that I have learned about the repository_dispatch event, there is also an implementation path for automating the build cycle of the distribution.