ESMValCore v2.11.0 release

[chrisbillowsMO]

Dear @ESMValGroup/esmvaltool-developmentteam,

Starting in a few weeks, and in the run up to the workshop, the Met Office will be preparing the release of ESMValTool v2.11.0, starting with the feature freeze of ESMValCore on APRIL 22ND, 2024.

Stay tuned for further news and updates here.

Manager	Buddy	Feature Freeze	Feature branch	Milestone
Met Office (@ehogan @chrisbillowsMO @mo-gill)	@bouweandela	2024-04-22

Minor Release	Status	Date	RTD	Release	conda-forge	PyPI	Docker
v2.11.0rc1		2024-05-13	Docs	v2.11.0rc1	esmvalcore_rc	esmvalcore (--pre)	docker
v2.11.0rc2		2024-06-27	Docs	v2.11.0rc2	esmvalcore_rc	esmvalcore (--pre)	docker
v2.11.0		2024-07-03	Docs	v2.11.0	esmvalcore	esmvalcore	docker

[valeriupredoi]

yay Core freeze on me birthday 🎂 Good luck, folks! Don't hesitate to ask us if stuck 👍

[chrisbillowsMO]

We're starting looking at this today! 😄

1. Check that all tests and builds work

• Check that the nightly test run on CircleCI was successful.
• Check that the GitHub Actions test runs were successful.
• Check that the documentation builds successfully on readthedocs.
• Check that the Docker images are building successfully.

All tests should pass before making a release (branch).

Additional Steps

• To access the information required at the Docker link, the user must be signed in and have access to the esmvalgroup organisation.

[schlunma]

I think that was just a hiccup of the CI machine. I restarted it, and it ran successfully 👍

[bouweandela]

Would it be worth considering moving entirely over to GitHub Actions and deprecating CircleCI?

Circle CI is better than GitHub actions in many respects (e.g. faster and bigger machines, better overview of tests) so I don't really see the point of moving, but we can discuss it at the next tech lead team meeting if you want. However, if you have a bit of time for updating infrastructure, maybe you could help us modernize our configuration by moving it from setup.py and setup.cfg to pyproject.toml?

[chrisbillowsMO]

if you have a bit of time for updating infrastructure, maybe you could help us modernize our configuration by moving it from setup.py and setup.cfg to pyproject.toml?

Great, I will definitely have a go at this. I've raised an issue on ESMValTool here.

[ehogan]

A brief update to include some notes from the TLT team meeting earlier this week:

The reason we have both CircleCI and GitHub Actions is that it has historically grown like this. CircleCI is better because it is a dedicated CI provider, but GitHub Actions is nice because it provides many predefined workflows.

It may be possible in the future to consolidate all workflows into CircleCI, but the benefits do not currently outweigh the development costs, and there are other, more impactful tasks that could instead be accomplished, as discussed above 😊

[bouweandela]

Here is another thing that could be done if you have some time for technical improvements: The @ESMValGroup/technical-lead-development-team agreed some time ago to try out towncrier instead of the draft_release_notes.py script because 1) the current changelog is not great for users, 2) difficult to populate for release managers who are not up-to-date with everything that happened in the project over the past few months, and 3) we would like to avoid maintaining custom software for writing release notes.

[chrisbillowsMO]

2. Create a release branch

• Create a branch off the main branch and push it to GitHub.
• Ask someone with administrative permissions to set up branch protection rules for it so only you and the person helping you with the release can push to it.

Additional Steps

• Check in with assignees for all outstanding PRs and Issues in the current milestone.

[chrisbillowsMO]

cc @ehogan

Hi @bouweandela - just two more on the release branch, please.

• Would we be right in thinking we should name the release branch v2.11.x in the style of this PR?
• There are still 14 open issues/PRs for this milestone - should we wait until these are merged before we make the release branch? What else should we do?

[valeriupredoi]

• yes
• you can create the release branch, and keep it updated with the latest main until the most strict freeze date ie when you stop accepting bugfixes: bugfixes can still be accepted past the soft freeze date, but no other developments should be - so in thie case a whole lot of other stuff gets merged past the freze date, a simple merge of main into the release branch won't do it good, you'd have to cherrypick the bugfix commits into it; best way would be to start chasing the people with PRs that you consider should be merged as part of 2.11 milestone, and ask them to hustle up

[bouweandela]

There are still 14 open issues/PRs for this milestone - should we wait until these are merged before we make the release branch? What else should we do?

Usually, the release manager checks in with people who have something in the milestone by commenting on their pull request 1 or 2 weeks before the feature freeze and asks them if they think they can finalize things before the feature freeze and helps to push a few things over the finish line that are almost there. I would not recommend waiting for everything because then the release will never happen.

[chrisbillowsMO]

Created the release branch from the tag as detailed in the discussion on "3. Increase the version" number below.

[chrisbillowsMO]

3. Increase the version number

The version number is automatically generated from the information provided by git using setuptools-scm, but a static version number is stored in CITATION.cff.

• Make sure to update the version number and release date in CITATION.cff. (See https://semver.org/ for more information on choosing a version number).
• Make a pull request
• Get it merged into main
• Cherry pick it into the release branch Not necessary as we did step No. 2 after this step.

[bouweandela]

a) Unfortunately citation.cff needs to be updated for every release and release candidate, because Zenodo uses the information: https://zenodo.org/records/11186676. Not updating it would mean the wrong entry is created on Zenodo.

b) The changelog is really useful to inform the ESMValTool recipe maintainers about the changes in ESMValCore. They can use this information to fix issues with their diagnostics, should that be needed. Therefore it is important that the changelog is available on readthedocs before updating environment.yml in the ESMValTool repository (as described in step 2 of https://docs.esmvaltool.org/en/latest/community/release_strategy/release_strategy.html#detailed-timeline-steps).

[chrisbillowsMO]

Oh wow - that Zenodo automation is awesome! How/when does it happen? We had a look in the GitHub workflows and found the PyPI one but couldn't see a specific Zenodo one?

Okay, understood on both points, will revert to the previous diagram! Thanks for confirming.

[bouweandela]

Zenodo automation is awesome! How/when does it happen?

It works like this: https://docs.github.com/en/repositories/archiving-a-github-repository/referencing-and-citing-content

[chrisbillowsMO]

Oh fascinating, thanks!

Re: the diagram - here's another pass with citatation.cff and changelog updated for each release.

We've added some extra detail on testing and some extra colours, key items etc. What do you think?

We're going to include this in our presentation on Monday (😨!!) so if you have further thoughts or comments please do let us know.

Not just if it's factually wrong but also if anything could be clearer - the key, layout, colours etc. Or you just happen to hate purple etc! All thoughts welcome! 😁

[bouweandela]

It looks very nice, good luck with the presentation!

[valeriupredoi]

FYI @chrisbillowsMO @ehogan we've started seeing random/flaky test fails on OSX with pip install, see ESMValGroup/ESMValTool#3580 - don't worry when/if you see this sort of failure during your release checks 👍

[chrisbillowsMO]

4. Add release notes

• Use the script esmvaltool/utils/draft_release_notes.py to create create a draft of the release notes.
• Open a discussion to allow members of the development team to nominate pull requests as highlights. (Here)
• Add the most voted pull requests as highlights at the beginning of changelog.
• After the highlights section, list any backward incompatible changes that the release may include.
• The backward compatibility policy. lists the information that should be provided by the developer of any backward incompatible change.
• Make sure to also list any deprecations that the release may include, as well as a brief description on how to upgrade a deprecated feature.
• Review the results, and if anything needs changing, change it on GitHub and re-run the script until the changelog looks acceptable.
• Copy the result to the file doc/changelog.rst.
• Make a pull request
• Get it merged into main
• Cherry pick it into the release branch.

---

• Move the following PR to Use iris.FUTURE.save_split_attrs = True to remove iris warning #2398 (comment) not backwards compatible

[bouweandela]

If it's just the date it's probably not worth the effort.

[chrisbillowsMO]

Issue and PR here: #2413 / #2414

[chrisbillowsMO]

Link to rendered changelog here

[ehogan]

See also:

• Update the version number and release notes for v2.11.0rc2 #2461
• Update the version number and release notes for v2.11.0rc2 #2462

[ehogan]

See also:

• Update the release date and version number for v2.11.0 #2473
• Update the release date and version number for v2.11.0 #2474

[chrisbillowsMO]

5. Make the (pre-)release on GitHub

• Do a final check that all tests on CircleCI and GitHub Actions completed successfully.
• Then click the releases tab and create the new release from main (see below) and create the new release from the release branch (i.e. not from main).
• Create a tag and tick the This is a pre-release box if working with a release candidate.

[chrisbillowsMO]

[ ] Then click the releases tab and create the new release from the release branch (i.e. not from main).

@bouweandela, would you confirm, following your code snippets (and our new diagram) that we are tagging main?

[bouweandela]

Yes

[chrisbillowsMO]

6. Mark the release in the main branch

• When the (pre-)release is tagged, it is time to merge the release branch back into main.

[chrisbillowsMO]

@bouweandela Are we right in thinking this step also isn't needed - if our previous assumptions are correct?

[bouweandela]

Yes, that could be deleted

[chrisbillowsMO]

7. Create and upload the PyPI package

• The package is automatically uploaded to the PyPI by a GitHub action.

[chrisbillowsMO]

8. Create the Conda package

• The esmvalcore package is published on the conda-forge conda channel.
• This is done via a pull request on the esmvalcore-feedstock repository.
• To publish a release candidate, you have to open a pull request yourself. An example for this can be found [here]Pre release v2.5.0rc6 conda-forge/esmvalcore-feedstock#35).
• Make sure to use the rc branch as the target branch for your pull request.
• Follow all instructions given by the linter bot.
• The testing of ESMValTool will be performed with the published release candidate.

[chrisbillowsMO]

For reference the feedstock PR is here.

[chrisbillowsMO]

9. Check the Docker images

After making the release, check that the Docker image for that release has been built correctly
by:

• checking that the version tag is available on Dockerhub
• checking the stable tag has been updated
• running some recipes with the stable tag Docker container, for example one recipe for Python, NCL, R, and Julia,
• running a recipe with a Singularity container built from the stable tag.

[bouweandela]

checking the stable tag has been updated

I noticed that there was a mistake in the Docker build configuration that triggered the stable build for the release candidate. I update the configuration so it now only builds for normal releases.

[chrisbillowsMO]

Hi @bouweandela - could we get some guidance on the last two points above please. How do we run recipes on the Docker/Singularity containers without access to ESMValTool? Or should the last two points only been done with the ESMValTool docker image (as part of the ESMValTool release)?

[bouweandela]

Docker instructions are available here: https://docs.esmvaltool.org/en/latest/quickstart/installation.html#docker-installation
Singularity instructions are available here: https://docs.esmvaltool.org/en/latest/quickstart/installation.html#singularity-installation

Docker containers are built every time a commit is pushed to the ESMValTool main branch. Therefore I would suggest you

1. make an ESMValCore release candidate that you are happy for the ESMValTool developers to use
2. open a pull request in ESMValTool to update environment[_osx].yml so it uses the ESMValCore release candidate
   3a) build the docker image locally and test: https://docs.esmvaltool.org/en/latest/community/release_strategy/release_strategy.html#check-the-docker-images
   3b) Alternatively: Get the pull requests merged and use the Docker container from Dockerhub esmvalgroup/esmvaltool:latest.

[chrisbillowsMO]

11. Run the recipes - step 2.3/2.4

• Uncomment the release candidate channel item (i.e. conda-forge/label/esmvalcore_rc) in the environment.yml of ESMValTool to add it to the list of channels used.
• Adjust the pin on ESMValCore after each release candidate (e.g. esmvalcore==2.8.0rc1).
• Check that the environment creation of ESMValTool works fine and contains the latest release candidate version.
• Run all the recipes (optionally with a reduced amount of data) to check that they still work with the release candidate.

Additional Steps

• Ensure instructions for installing Julia dependencies in the environment are followed

[chrisbillowsMO]

See the test result issue here:

• Recipe test results for ESMValCore v2.11.0rc1 #2421

[ehogan]

Steps I actually ran on DKRZ:

Install miniforge (mamba) on DKRZ

• Download miniforge install script: mkdir ~/downloads; cd ~/downloads; curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh"
• Install miniforge: bash Miniforge3-$(uname)-$(uname -m).sh
• Update mamba and conda: mamba update --name base mamba conda

Create an ESMValTool environment containing the release candidate

• Setup SSH keys
• Clone ESMValTool: mkdir ~/software; cd ~/software; git clone git@github.com:ESMValGroup/ESMValTool.git; cd ESMValTool
• Edit the ESMValTool environment.yml file to include the ESMValCore release candidate (details above)
• Create the environment: mamba env create --name esmvaltool-core2.11.0rc1 --file environment.yml
• Activate the ESMValTool environment: conda activate esmvaltool-core2.11.0rc1
• Install ESMValTool into the ESMValTool environment: pip install .
• Install Julia dependencies: esmvaltool install Julia

Configure ESMValTool

• Create an ESMValTool user configuration file from the ESMValTool environment containing the release candidate: esmvaltool config get_config_user
• Update the ESMValTool user configuration file (located in ~/.esmvaltool/config-user.yml):
  • set output_dir to /scratch/b/<username>/run1/esmvaltool_output
  • ensure log_level is set to info
  • ensure remove_preproc_dir is set to true
  • uncomment the DKRZ entries

Run recipes

• Update the esmvaltool/utils/batch-jobs/generate.py script, ensure submit is set to False
• Create a new directory for the batch scripts: mkdir -p ~/testing_v2.11.0rc1/run1/batch_scripts; cd ~/testing_v2.11.0rc1/run1/batch_scripts
• Generate the batch scripts: python ~/<path>/<to>/esmvaltool/utils/batch-jobs/generate.py
• Submit the batch scripts by setting submit to True and re-generating the batch scripts
• Copy the generated dask.yml file to ~/.config/dask/dask.yml
• Rerun specific recipes (TBD)

[chrisbillowsMO]

12. Make another release candidate - step 2.5 to 2.7

• If a bug is discovered that needs to be fixed before the release, a pull request can be made to the main branch to fix the bug.
• The person making the pull request can then ask the release manager to cherry-pick that commit into the release branch.
  • RM asked to include by @valeriupredoi : Pin numpy<2.0.0 to avoid pulling 2.0.0rcX #2415
  • RM asked to include by @bouweandela: Load data only once for ESMPy regridders #2418
  • RM asked to include by @ehogan: Fix bug in _rechunk_aux_factory_dependencies #2428
  • RM asked to include by @bouweandela: Avoid loading entire files into memory when downloading from ESGF #2434
  • RM asked to include by @bouweandela: Add support for Python=3.12 #2228
  • RM asked to include by @alistairsellar: Add native6, OBS6 and RAWOBS rootpaths to metoffice config-user.yml template, and remove temporary dir #2432
  • RM asked to include by @bouweandela: Preserve cube attribute global vs local when concatenating #2449
  • RM asked to include by @schlunma: Fix units and valid_range of AIRS-2-0 #2465
• Make another release candidate including the bugfix(es). We are skipping this in favour of the below per the Slack comments:
• Do a from source installation of both ESMValTool and ESMValCore and then run the tests.

I would keep the same environment for all test runs, just keep pulling in new versions of the release branch(es).

It may be a bit cleaner to only test with release candidates (I'm not even sure if it is), but on the other hand, testing every recipe with every release candidate is quite wasteful of computational resources.

• Run the affected recipes again to check for further bugs.
• Make as many release candidates for ESMValCore as needed in order to fix all the detected bugs.

[chrisbillowsMO]

#2449 cherry picked in here - June 13th commit.

[ehogan]

#2465 cherry picked, see "Jun 26, 2024" commit on the v2.11.x release branch 😊

[bouweandela]

Don't forget to include #2462.

Since ESMValGroup/ESMValTool#3628 is included in the ESMValTool release, it would be nice to add #2464 to the corresponding ESMValCore release, as this makes sure the first page you see when you go to https://docs.esmvaltool.org and then click on ESMValCore related stuff and then go back to the first page looks the same.

[ehogan]

Thanks @bouweandela 🎉 #2462 and #2464 have been cherry picked, see "Jun 27, 2024" commit on the v2.11.x release branch 😊

[ehogan]

• Update the version number and release notes for v2.11.0rc2 #2462 cherry picked, see "Jul 3, 2024" commit on the v2.11.x release branch 😊

[valeriupredoi]

excellent work, folks 👏 🍺 🎖️

[bouweandela]

@ehogan @chrisbillowsMO Now that the release is out, were you still planning on updating the documentation with this very nice figure #2389 (reply in thread) and the new procedure which avoids merging back the release branch into the main branch? I'm sure that would make the life of future release managers a lot easier.

[ehogan]

Yes 😁 We have an issue for this (#2399) and I will work on this in the coming weeks 👍