Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support publishing preview releases #19

Merged
merged 1 commit into from
Jan 9, 2024
Merged

Conversation

rtyley
Copy link
Member

@rtyley rtyley commented Jan 4, 2024

This addresses issue #10, providing support for publishing preview releases based off feature branches, ie Pull Requests!

It doesn't provide SNAPSHOT releases, but hopefully preview releases should serve most of the use cases.

UX

The UX of doing a preview release is very similar to the existing process for doing a full release, with only these differences:

  • The developer needs to select the PR branch before clicking the green Run workflow button
  • The version number will be the upcoming version number, but with a suffix that clearly indicates this is a preview release, eg: 1.0.7-PREVIEW.feature1.2024-01-04T1230.42ed11d4.
  • No branches are updated by the release (ie, not the PR's feature branch, and not the default main branch) - the preview release commit exists as its own tagged commit, taking the latest PR commit as its parent.
  • GitHub release notes will not be created, but instead the PR using that branch will be updated with a comment providing details of the new release (version number, etc):
    image

Internal changes

Internally, these changes take place if a non-default branch (ie a feature branch, not main) is used:

  • The 🔒 Init job release_type output is PREVIEW_FEATURE_BRANCH rather than FULL_MAIN_BRANCH
  • Only 1 commit is pushed by the workflow, rather than 2, and not onto the branch - the single commit exists as a tagged leaf to the side of the PR branch. The 2nd commit, normally needed by the full release process (incrementing the version number and adding the -SNAPSHOT suffix), is not needed for preview releases - there are already enough details in the -PREVIEW... version-suffix to keep preview releases unique, even if you publish many releases for 1 commit in 1 PR.
  • When that 1 commit is pushed, it's initially pushed with a disposable Git tag - not the release tag. Pushing any commit requires either a branch or tag for the git push command to work on (you can't just push a commit id - I've tried), and there is no pre-existing suitable branch (we don't want to modify the PR feature branch) or tag (the release tag has an annotation message including the hashes of all artifact files generated by the release, and at the point when the commit is pushed, those artifacts & their hashes are not available yet), so we have to use a new, different, disposable, Git tag instead.

runs-on: ubuntu-latest
permissions:
contents: write
pull-requests: write
Copy link
Member Author

@rtyley rtyley Jan 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In order for the workflow to comment on PRs (notifying that a preview has been published), we need the pull-requests: write permission.

This also means updating all the release.yml files in the projects currently using gha-scala-library-release-workflow, so that they look like this:

    permissions: { contents: write, pull-requests: write }

...I've now done this, as you can see in guardian/etag-caching#33 and the associated commits on all the other repos.

@rtyley rtyley force-pushed the support-preview-releases branch 2 times, most recently from 10950c6 to c7f56f8 Compare January 4, 2024 10:46
rtyley added a commit that referenced this pull request Jan 4, 2024
@rtyley rtyley changed the title Support preview releases Support publishing preview releases Jan 4, 2024
@rtyley rtyley changed the title Support publishing preview releases Support publishing PREVIEW releases Jan 4, 2024
@rtyley rtyley force-pushed the support-preview-releases branch 2 times, most recently from f300d2f to ddfbda2 Compare January 4, 2024 11:00
version_suffix=""
else
release_type="PREVIEW_FEATURE_BRANCH"
version_suffix="-PREVIEW.${GITHUB_REF_NAME//[^[:alnum:-_]]/}.$(date +%Y-%m-%dT%H%M).${GITHUB_SHA:0:8}"
Copy link
Member Author

@rtyley rtyley Jan 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lots of stuff going on in the version_suffix! :

  • Starting - hyphen - https://semver.org/#spec-item-9 says "9. A pre-release version MAY be denoted by appending a hyphen ..."
  • PREVIEW - this is recognised by IntelliJ & Scala Steward to mean 'non-stable-release, not suitable for auto-upgrading to', which is what we want 👍
  • ${GITHUB_REF_NAME//[^[:alnum:-_]]/} - this is the PR branch name, with most non-alphanumeric characters stripped out - in particular, stripping out slash (/), which some people use in Git branch names but which we couldn't have as part of the version, because Maven uses the version as part of the artifact url path.
  • $(date +%Y-%m-%dT%H%M) - this is the date, to the minute (eg 2024-01-04T1230), helping to differentiate and order mulitple preview releases for one PR.
  • ${GITHUB_SHA:0:8} - this is the first 8 digits of the hash id for the PR commit that the preview release was based on. This helps to differentiate and identify multiple preview release for one PR, but also is recognised by Scala Steward, again meaning 'non-stable-release, not suitable for auto-upgrading to', which is what we want 👍

Choice of version-suffix for preview releases

https://semver.org/#spec-item-9 says:

  1. A pre-release version MAY be denoted by appending a hyphen and a series of dot separated identifiers immediately following the patch version. Identifiers MUST comprise only ASCII alphanumerics and hyphens [0-9A-Za-z-]. Identifiers MUST NOT be empty. Numeric identifiers MUST NOT include leading zeroes. Pre-release versions have a lower precedence than the associated normal version. A pre-release version indicates that the version is unstable and might not satisfy the intended compatibility requirements as denoted by its associated normal version. Examples: 1.0.0-alpha, 1.0.0-alpha.1, 1.0.0-0.3.7, 1.0.0-x.7.z.92, 1.0.0-x-y-z.--.

  2. Build metadata MAY be denoted by appending a plus sign and a series of dot separated identifiers immediately following the patch or pre-release version. Identifiers MUST comprise only ASCII alphanumerics and hyphens [0-9A-Za-z-]. Identifiers MUST NOT be empty. Build metadata MUST be ignored when determining version precedence. Thus two versions that differ only in the build metadata, have the same precedence. Examples: 1.0.0-alpha+001, 1.0.0+20130313144700, 1.0.0-beta+exp.sha.5114f85, 1.0.0+21AF26D3----117B344092BD.

How can we prevent tooling from thinking our preview releases are stable releases?

If we're not using the -SNAPSHOT suffix, there's a risk that tooling will assume that our preview releases are stable releases, and attempt to auto-upgrade to them.

Consequently, to be certain of being recognised as a pre-release, it seems wise to include these components in the version number:

  • 'PREVIEW'
  • a commit hash of at least 8 characters

NPM version numbers...

Some Guardian libraries are released simultaneously for both Scala and other platforms like NPM (for instance, content-api-models, see guardian/content-api-models#229).

Both NPM and sbt/Maven work well with simple three-number semver version numbers, but how well will NPM handle extended version number like 1.0.7-PREVIEW.feature1.2024-01-04T1230.42ed11d4 ?

For the time being, I'm happy enough to go ahead with this version-number structure, but alter it later if it causes problems.

'PREVIEW', 'BETA', 'ALPHA', or...?

Justin points out the 'PREVIEW' has a particular meaning for the Content Pipeline team ('Preview' vs 'Live' content) - could potentially cause some confusion.

I initially chose 'PREVIEW' (from the identifiers that IntelliJ & Scala Steward understand) partially because both 'BETA' & 'ALPHA' imply some meaning about the stage of development in the software release
cycle that may or may not be appropriate - it's not obvious which one truly reflects what we're doing when we make an early release from a PR, and it's annoying to have to choose. Additionally, looking at
the Wikipedia article that describes the different stages - the label 'pre-alpha' might even be more appropriate: https://en.wikipedia.org/wiki/Software_release_life_cycle

@@ -143,12 +173,24 @@ jobs:

cat << EndOfFile >> $GITHUB_OUTPUT
release_tag=$RELEASE_TAG
release_version=${RELEASE_TAG#"v"}
Copy link
Member Author

@rtyley rtyley Jan 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here we're using Bash Shell Parameter Expansion to strip the v off the front of the release tag (eg v1.0.6-PREVIEW.feature1.2023-12-31T1334.fd006d63) to get the release version (which is what you'd use when declaring the library as a dependency, eg 1.0.6-PREVIEW.feature1.2023-12-31T1334.fd006d63).

The RELEASE_VERSION is used later on in the workflow in the PR comment, as that's probably what people want, rather than the tag:

image

@rtyley rtyley force-pushed the support-preview-releases branch 6 times, most recently from cf021fa to 6d51728 Compare January 4, 2024 17:35
This addresses #10,
providing support for publishing *preview* releases based off feature branches, ie PRs.

The UX of doing a preview release is very similar to the existing process for doing a full
release (https://github.com/guardian/gha-scala-library-release-workflow/blob/main/docs/making-a-release.md),
with only these differences:

* The developer needs to select the PR branch before clicking the green `Run workflow` button
* The version number will be the **upcoming** version number, but with a suffix that clearly
  indicates this is a preview release, eg: `1.0.7-PREVIEW.feature1.2024-01-04T1230.42ed11d4`.
  Note that this is _not_ a `-SNAPSHOT` release, the workflow does not support `SNAPSHOT`
  releases.
* No branches are updated by the release (ie, not the PR's feature branch, and not the
  default `main` branch) - the preview release commit exists as its own tagged commit,
  taking the latest PR commit as its parent.
* GitHub release notes will not be created, but instead the PR using that branch will be
  updated with a comment providing details of the new release (version number, etc).

Internally, these 'preview release' changes take place if a non-default branch (ie a feature
branch, not `main`) is used:

* The `🔒 Init` job `release_type` output is `PREVIEW_FEATURE_BRANCH` rather than `FULL_MAIN_BRANCH`
* Only 1 commit is pushed by the workflow, rather than 2, and _not_ onto the branch - the
  single commit exists as a tagged leaf to the side of the PR branch. The 2nd commit normally needed
  by the full release process (incrementing the version number and adding the `-SNAPSHOT`
  suffix) is not needed for preview releases - there are already enough details in the `-PREVIEW`
  version-suffix to keep preview releases unique, even if you do many releases for 1 commit in 1 PR.
* When that 1 commit is pushed, it's initially pushed with a _disposable_ Git tag - not the
  _release_ tag. Pushing _any_ commit requires either a branch or tag for the `git push` command
  to work on (you can't just push a commit id - I've tried), and there is no pre-existing suitable
  branch (we don't want to modify the PR feature branch) or tag (the release tag has an annotation
  message including the hashes of all artifact files generated by the release, and at the point
  when the commit is pushed, those artifacts & their hashes are not available yet), so we have to use
  a new, different, disposable, Git tag instead.

## Choice of version-suffix for preview releases

https://semver.org/#spec-item-9 says:

> 9. A pre-release version MAY be denoted by appending a hyphen and a series of dot separated identifiers immediately following the patch version. Identifiers MUST comprise only ASCII alphanumerics and hyphens [0-9A-Za-z-]. Identifiers MUST NOT be empty. Numeric identifiers MUST NOT include leading zeroes. Pre-release versions have a lower precedence than the associated normal version. A pre-release version indicates that the version is unstable and might not satisfy the intended compatibility requirements as denoted by its associated normal version. Examples: 1.0.0-alpha, 1.0.0-alpha.1, 1.0.0-0.3.7, 1.0.0-x.7.z.92, 1.0.0-x-y-z.--.
>
> 10. Build metadata MAY be denoted by appending a plus sign and a series of dot separated identifiers immediately following the patch or pre-release version. Identifiers MUST comprise only ASCII alphanumerics and hyphens [0-9A-Za-z-]. Identifiers MUST NOT be empty. Build metadata MUST be ignored when determining version precedence. Thus two versions that differ only in the build metadata, have the same precedence. Examples: 1.0.0-alpha+001, 1.0.0+20130313144700, 1.0.0-beta+exp.sha.5114f85, 1.0.0+21AF26D3----117B344092BD.

### How can we prevent tooling from thinking our preview releases are stable releases?

If we're not using the `-SNAPSHOT` suffix, there's a risk that tooling will assume that our preview releases are stable releases, and attempt to auto-upgrade to them.

* IntelliJ automatically suggests dependency upgrades - it uses `PackageVersionNormalizer` with specific stability tokens that include 'preview'
* Scala Steward raises dependency upgrade PRs - it uses `isPreRelease` which recognises `Hash` (6+ or 8 hex chars) & specific `Alpha` components that include 'preview'. See also scala-steward-org/scala-steward#1033, scala-steward-org/scala-steward#1549 etc
* Scaladex uses PreRelease.scala - scalacenter/scaladex#614

Consequently, to be certain of being recognised as a pre-release, it seems wise to include these components in the version number:

* 'PREVIEW'
* a commit hash of at least 8 characters

### NPM version numbers...

Some Guardian libraries are released simultaneously for both Scala and other platforms like NPM
(for instance, `content-api-models`, see guardian/content-api-models#229).
Both NPM and sbt/Maven work well with simple three-number semver version numbers, but how well will
NPM handle extended version number like `1.0.7-PREVIEW.feature1.2024-01-04T1230.42ed11d4` ?

### 'PREVIEW', 'BETA', 'ALPHA', or...?

Justin points out the 'PREVIEW' has a particular meaning for the Content Pipeline team ('Preview' vs 'Live'
content) - could potentially cause some confusion there.

I initially chose 'PREVIEW' (from the identifiers that IntelliJ & Scala Steward understand) partially
because both 'BETA' & 'ALPHA' imply some meaning about the stage of development in the software release
cycle that may or may not be appropriate - it's not obvious which one truly reflects what we're doing
when we make an early release from a PR, and it's annoying to have to choose. Additionally, looking at
the Wikipedia article that describes the different stages - the label 'pre-alpha' might even be more
appropriate: https://en.wikipedia.org/wiki/Software_release_life_cycle

## Problems with backticks

Ideally we would be generating a markdown message for the PR comment with lots of backticks
for styling:

https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/creating-and-highlighting-code-blocks

...like this:

rtyley/sample-project-using-gha-scala-library-release-workflow#1 (comment)

...but they get interpreted by BASH, and cause trouble...

https://github.com/rtyley/sample-project-using-gha-scala-library-release-workflow/actions/runs/7399435634/job/20130944058

...so for the time being this PR avoids them in the generated PR comment.

cat comment_body.txt

gh pr comment ${{ github.ref_name }} --body-file comment_body.txt >> $GITHUB_STEP_SUMMARY
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Problems with Bash interpreting backticks have meant that the comment message isn't quite as pretty as I'd like - ideally there would be backticks for styling code blocks, like this:

image

For the time being this PR avoids them in the generated PR comment.

@rtyley rtyley marked this pull request as ready for review January 4, 2024 18:31
@rtyley rtyley changed the title Support publishing PREVIEW releases Support publishing preview releases Jan 4, 2024
@rtyley rtyley requested a review from ioannakok January 5, 2024 11:57
rtyley added a commit to guardian/etag-caching that referenced this pull request Jan 5, 2024
guardian/gha-scala-library-release-workflow#19
introduces support for publishing preview releases, but note this does
use the `pull-requests: write` permission in order to comment on the PR
to announce that the release has been published - so we need to
additionally grant that permission:

https://github.com/guardian/gha-scala-library-release-workflow/pull/19/files#r1441589383
@rtyley rtyley merged commit 6d93a92 into main Jan 9, 2024
@rtyley rtyley deleted the support-preview-releases branch January 9, 2024 16:03
rtyley added a commit to rtyley/sample-project-using-gha-scala-library-release-workflow that referenced this pull request Jan 9, 2024
Now that guardian/gha-scala-library-release-workflow#19 is merged, we don't need to be using this branch of the workflow anymore.
rtyley added a commit to guardian/mobile-apps-api-models that referenced this pull request Jan 10, 2024
Since guardian/gha-scala-library-release-workflow#19, we've been able to do preview releases too!
@rtyley
Copy link
Member Author

rtyley commented Jan 17, 2024

One update to this - Scaladex badges were failing to ignore pre-releases (even though the Scaladex does have the concept of a pre-release), but this has been fixed with scalacenter/scaladex#1330

rtyley added a commit to guardian/pan-domain-authentication that referenced this pull request Apr 26, 2024
The Maven badges we were using don't understand about pre-release version numbers, thankfully the Scaladex badges do!

See also:

* guardian/gha-scala-library-release-workflow#19 (review)
* scalacenter/scaladex#1330
rtyley added a commit that referenced this pull request Jun 25, 2024
This should fix #37 - ie, a Full Main-Branch release should still succeed, even if someone adds
additional commits to the `main` branch while the release workflow is running.

Note that this change in behaviour only applies to Full Main-Branch releases, not Preview releases
(#19), as Preview releases don't
ever attempt to update any branch.

To avoid the noise of an additional merge commit (ie a 3rd commit) we prefer a fast-forward, and
only create a merge commit if it's _necessary_ - ie because we have to merge on top of some commits
that someone else has added.

## Two full Main-Branch releases can't run concurrently

Although we can now handle extra work on the `main` branch, running **two releases at once** is _not_ recommended!

## Using the GitHub API to create & merge commits, rather than `git`

We're now using the GitHub API, rather than `git`, to do both these operations:

* [Update the branch](https://docs.github.com/en/rest/git/refs?apiVersion=2022-11-28#update-a-reference) to try fast-forwarding - using the GitHub API means we don't need to clone a deep history of the repo to make that update
* [Merge the branch](https://docs.github.com/en/rest/branches/branches?apiVersion=2022-11-28#merge-a-branch) if the fast-forward fails - using the GitHub API means that the merge commit will be signed/verified by GitHub too, [just like other commits created by the GitHub App](#26).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant