Skip to content

Releasing

Jump to bottom
Muhammad Hamza Sajjad edited this page Jul 1, 2024 · 21 revisions

Release Tooling

This document details the release process for ArcticDB. ArcticDB is released onto PyPi and conda-forge.

0. Check Internal Tests and Benchmark Results

Check Man Group's internal tests against the release candidate and benchmark. See "ArcticDB - Support Process" internal wiki page.

1. Create a new tag

Navigate to the Tag and Release Github Action.

Click Run Workflow on the right hand side.

Important

Release candidate, full release, or hotfix?

If preparing a release for a new major or minor version (SemVer versioning), create a pre-release version (X.Y.ZrcN).

If preparing a release for a new patch version, create a full release (X.Y.Z).

If there is an outstanding release candidate (look here) and no issues have been found, promote the release candidate to a full release (X.Y.Z).

Run ArcticDB/docs/mkdocs/docs/technical/release_checks.py against a release candidate before promoting it to a full release.

If creating a new release off master:

  1. Type in the new version number eg 1.6.0 or 1.6.0rc0.
  2. Click Run workflow.

If promoting a pre-release:

  1. Type in the new version number eg 1.6.0
  2. Select the workflow off the source tag (e.g. 1.6.0rc0)
  3. Click Run workflow

If creating a patch release:

  1. If this is the first patch for this version you must create a branch of the version tag named X.Y.x. For example, the patch branch for version 4.3.0 must be named 4.3.x and be branched out of tag v4.3.0. All further patch versions must be released using that branch.
  2. Make your patch changes and open a PR to merge into X.Y.z.
  3. Later when running running the Tag and Release workflow you must specify the X.Y.z branch in the use workflow from field.

The build will now be running for the tag.

2. Update conda-forge recipe

A new release on conda-forge is made by creating a Pull Request on the feedstock: conda-forge/arcticdb-feedstock.

For more information about conda-forge release process and feedstocks' maintenance, see this section of the documentation of conda-forge.

If you have created a tag for a release on ArcticDB's repository, regro-cf-autotick-bot might already have opened a Pull Request to create a new release. You can push commits to this PR or alternatively create a new PR to release a version.

Important

Do not commit directly to the feedstock repository. Commits to the repository release new versions to conda-forge. Instead, changes must be made via personal forks or via the PR created by the regro-cf-autotick-bot as mentioned above.

If publishing a normal release, you must create two pull-requests due to some users' constraints (see the description of the introduction of the libevent-2.1.10 branch in this PR for more context):

  • one pull request which branches from main and merge the created PR into the main branch.
  • one pull request which branches from libevent-2.1.10 and merge the created PR into the libevent-2.1.10 branch.

If publishing a release-candidate, you must branch from rc and merge the created PR into the rc branch.

Do the following for each PR:

  1. Fork arcticdb-feedstock (Untick the option to just copy main.)
  2. Create a branch of the base branch for your case (see above)
  3. Change recipe/meta.yaml to update the following pieces of information:
    • version, pointing to the tag created in step 1
    • sha256sum of the source tarball. In order to get this download the release tarball from GitHub. Then in a terminal (or GitBash on Windows) run: sha256sum ArcticDB-x.y.z.tar.gz.
    • the build number (i.e. number under the build section) to 0
    • the dependencies if they have changed since the last version in:
  4. Push on your fork and create a PR to the feedstock targeting the base branch on conda-forge/arcticdb
  5. Rerender the feedstock's recipe to create Azure CI jobs' specification for all variants of the package This can be done by posting a comment on the PR with the following content @conda-forge-admin, please rerender

Example pull-requests:

3. Release to PyPi

The GitHub Actions "Build and Test" job kicked off automatically at the end of step 1 will wait on approval to deploy to PyPi. Navigate to the Deployments -> ProdPypi. Find the deployment click the three dots -> View Workflow Run -> Publish -> View Pending Deployments and approve it.

If there is a problem with this final step of the workflow, e.g. with our node script, then it is possible to run the publish.yml workflow manually. Go to the Publish to Pypi workflow -> Run Workflow -> Set "Use workflow from" to your tag, "envrionment" to ProdPypi, and "run_id" as the number from the previous Build and Test workflow which failed under the publish step, e.g. 7210804495 for https://github.com/man-group/ArcticDB/actions/runs/7210804495/. This will publish the artifacts from the failed Build and Test workflow to PyPI.

4. Release to Conda

Merge the PR created in step 3.

It will build packages, pushing them to the cf-staging channel before publishing them on the conda-forge channel for validation.

Packages are generally available about half an hour after the CI completes on main.

You can search for your package to see if it has been published with:

mamba search 'conda-forge/label/arcticdb_rc::arcticdb==4.1.0rc0'  # RC
mamba search 'conda-forge::arcticdb==4.0.0'  # Normal

5. Create Branch for new Minors/Major

If this is new minor or major (e.g. 2.0.0) or a minor (e.g. 1.5.0) off-master create a branch X.Y.x off the release tag. This is to ensure that the next patches are on top of the previous version. You should do this for release candidates.

(This way if a release candidate (e.g. 1.5.0rc0) needs trial before promotion, we can fix potential bugs in the branch without having to pause non related dev merges to master.)

6. Update the BSL conversion table

If you are releasing either a major (e.g. 2.0.0) or a minor (e.g. 1.5.0) version, update the license conversion schedule table in the readme.

The conversion date will be two years from when the release is published on GitHub. This is not required if you are releasing a patch release.

7. Docs

To release the docs,

  • Create a tag v<x.y.z>-docs where <x.y.z> stands for Arctic's version
    • From the cmdline you could use git fetch --tags; git checkout v<x.y.z>; git tag v<x.y.z>-docs; git push origin --tags
  • Run the Docs Build action for your new tag.
    If this is the latest stable release of ArcticDB (i.e. not a backport) then also tick latest. Tick Push the built docs to the docs-pages branch on github box unless you are testing the docs build. Normally you would tick this, because the publish action reads from there.
  • Run the Docs Publish action with the Prod environment. The Publish action needs approval and will upload to https://docs.arcticdb.io (hosted by Cloudflare Pages).

See the Docs README for more information.

8. Release Notes

Only do this for normal releases, not release candidates.

After the PyPi release, draft release notes will appear here.

Edit them and take particular care to announce API changes and describe important new features and bugfixes. Share the draft internally to allow the PR submitter to amend if necessary. When satisfied, decide whether to tick "set as latest release", and press publish.

Then copy the notes and announce the release in Slack, in the ArcticDB #general channel and the Man Group #arcticdb-announcements channel.

Dealing with bad releases

Note - instructions in this section are only to be followed if a broken build has been released. This section can be ignored under normal circumstances.

Conda-forge

Conda-forge packages are immutable. As a result, packages cannot be removed entirely.

They can, however, be marked as broken. This adds a new label to the package, broken. Packages with the broken label must be installed via a special channel (conda install -c conda-forge/label/broken) and thus for most users won't be visible.

To mark an ArcticDB package as broken:

  1. Find all existing files for the given version
  2. Create a PR that adds a new file listing every file to be marked as broken. Here's an example.
  3. Wait for someone from the Core team to merge the PR.

Files marked as broken can be unmarked afterward.

Hotfix Process

If hotfixing an existing release (or pre-release) then branch off the previously-released tag, apply the necessary changes (cherry-picking from master if commits are on master), and run the "Tag and Release" job.

  1. Type in the new version number eg 1.6.1 (or 1.6.0rc2 as an example of a pre-release version)
  2. Select the workflow off the source tag (e.g. 1.6.0 or 1.6.0rc1)
  3. Click Run workflow.

Then follow the steps in the main part of this document for 2 downwards as usual.

ArcticDB Wiki

Clone this wiki locally