CI lockfiles #4108

jamesp · 2021-04-27T18:44:44Z

🚀 Pull Request

Description

Here's a proof of concept for adding locked dependencies and using them in the nox testing framework.

Lock files are found in requirements/ci/nox.lock. These are explicit conda environment files which means they do not require resolving, they are the resolved manifestation of the equivalent requirements/ci/py3X.yml. This means that they can be installed using conda create -n my_new_env --file requirements/ci/nox.lock/py3X-linux-64.lock.
noxfile.py updated to use these resolved environments and avoid rebuilding / updating until the lockfile changes. This should dramatically reduce the second time the nox test suite runs. It should probably also greatly reduce the time the first run takes too, as there is no need for conda to resolve or update the environment.
A github action workflow to generate new lockfiles every Saturday morning, and create a PR with the changed files. This can then be included/ignored/deleted at our leisure.

There's quite a lot in this, so whoever feels like reviewing, lets go through it together 👍

This should resolve #4105

Consult Iris pull request check list

Comments and cron scheduling for refresh-lockfiles workflow

noxfile now handles out of date environments, so no need to invalidate the cache in the cirrus configuration

jamesp · 2021-04-27T18:47:03Z

You can see an example of the GH Actions run here: https://github.com/jamesp/iris/actions/runs/790009248

And the PR it produces here: jamesp#3

bjlittle · 2021-04-28T09:09:11Z

@jamesp Did you try using conda-lock with mamba in miniforge to revolve in realtime?

Or would that also struggle in cirrus-ci?

bjlittle · 2021-04-28T09:14:06Z

We could also rationalise the requirements themselves, as conda-lock can resolve multiple yml files at once, which is super handy. i.e., the testing jobs don't need the documentation requirements, and I suspect that the minimal testing jobs don't need the optional requirements.

There are options to make this less bloaty, package wise... but maybe that can come later.

jamesp · 2021-04-28T09:15:19Z

No I didn't but I'm sure that would work too and would be a viable alternative to this setup. But then you are still doing a resolve on every run of the CI.

We first need to decide if lockfiles belong in version control. If yes, this scheme provides a mechanism for keeping them up to date. If no, this scheme should be discarded. we could instead keep the lockfiles cached in CI to reduce the number of resolves

trexfeathers · 2021-04-28T09:17:27Z

We first need to decide if lockfiles belong in version control. If yes, this scheme provides a mechanism for keeping them up to date. If no, this scheme should be discarded. we could instead keep the lockfiles cached in CI to reduce the number of resolves

I'm 100% in favour of the former and against the latter. As I said before, this can solve issues beyond CI - benchmarking being the example I'm aware of currently.

jamesp · 2021-04-28T09:19:21Z

The pro for this approach is that we get a known, reproducible, test environment for every CI run until we explicitly update the lockfiles committed to git.

As a workflow, I was thinking somethign along the lines of:

GH Action updates the lockfiles once a week and creates a PR. If all tests pass, we merge the PR and now tests run against latest dependencies.
If tests fail on this PR (most likely being matplotlib image hash discrepancies) we CO the PR, fix the code & tests, commit and merge. Changes required due to dependency change are therefore isolated from other development changes.

bjlittle · 2021-04-28T09:21:57Z

@jamesp How will this work for feature branches, that pull in additional or different requirements. Any thoughts?

bjlittle · 2021-04-28T09:23:09Z

@jamesp Interesting that there are graphical phash failures... what's changed?

Do you have scope and capacity to investigate this further?

rcomer · 2021-04-29T11:05:12Z

there still seems to be an imagehash failure

That particular gallery example is missing bits (see #4110). I have no idea if the failure is related to that though.

Edit: now thinking it quite likely is that: #4110 (comment)

jamesp · 2021-04-29T11:46:07Z

What do you want to do about the link-check failures, given we know there is a fix in #4104?

Given the output of the test above suggests the code is working, it's the same test that's failing,I would suggest, if everyone is comfortable, we ignore that here, and rebase #4104.

I'm not sure about the gallery failure, I'll take a look at the imagehash on 3.8 and see what the issue is. the hamming distance score suggests it's not far off.

rcomer · 2021-04-29T12:10:39Z

@jamesp I think a change at matplotlib v3.4 has reinstated correct behaviour for that gallery example. See #4110

jamesp · 2021-04-29T12:33:35Z

We should address that image hash in a separate PR, I think this one is about complete now.

jamesp · 2021-05-04T09:09:18Z

What more do we need to do to get comfortable with merging this with the failing doc tests? We seem to be approaching a circular dependency across a number of PRs!

trexfeathers · 2021-05-04T09:12:50Z

What more do we need to do to get comfortable with merging this with the failing doc tests? We seem to be approaching a circular dependency across a number of PRs!

@pp-mo also suggested we should briefly discuss this as a team. It's not just a matter of "does it work?" - it's a change in how we do things.

trexfeathers · 2021-05-04T09:14:56Z

@jamesp don't get me wrong, I'm also very keen we get this over the line before we get an even bigger spider web of blocking problems.

Picked from SciTools@566f187

@trexfeathers

* 'ci-lock' of github.com:jamesp/iris: Apply suggestions from @trexfeathers code review

jamesp · 2021-05-04T09:42:28Z

Thanks @trexfeathers. Pushed commits to address those points above.

I agree this is a change to how things are done that should be discussed. Could @trexfeathers or @pp-mo setup a quick call for all those interested?

rcomer · 2021-05-04T10:16:06Z

That anomaly log gallery failure also came up in my local testing but I ignored it because it wasn't previously failing here. The idiff output looked similar to the change shown at SciTools/test-iris-imagehash#40, except the colour bar was also affected.

For the square root failure, see #3993.

😔

jamesp · 2021-05-04T14:01:12Z

I've reverted the changes to imagerepo.json as they should be in a separate PR. Let's keep this one entirely focussed on the CI changes so that it can be easily reverted if needed.

Once this is merged as a priority we should fix:

square root failure Failing test_square_root test #3993
image hashes for galleries and test
linkcheck failures.

trexfeathers

Can confirm this only includes the expected CI failures. Merging as agreed.

* master: (49 commits) Update CI environment lockfiles + Cartopy 0.19 changes (SciTools#4125) separate arg types from descriptions (SciTools#4100) Use assertArrayAllClose for sqrt test (SciTools#4118) Removed branch suffix (SciTools#4117) Corrected plot_anomaly_log_colouring for new Matplotlib linscale rules. (SciTools#4115) replace most recent hashes (SciTools#4112) Linkcheck update (SciTools#4104) CI lockfiles (SciTools#4108) Fix bug in gallery_tests runner (SciTools#4111) add logo to conda and pypi badges (SciTools#4088) pre-commit-ci update (SciTools#4092) Fix cirrus ci and mpl (SciTools#4087) Update readme docs stable (SciTools#4089) linux kernel version fix for conda 4.10+ (SciTools#4084) update docs pypi instructions (SciTools#4077) update flake8 pre-commit (SciTools#4067) Add GitHub discussions badge (SciTools#4070) conda requirements add pip (SciTools#4062) add pre-commit.ci badge and doc support (SciTools#4061) cirrus-ci credits for non-master (SciTools#4060) ...

* master: (87 commits) Contourf Antialias Fix (SciTools#4150) minor tidy of cirrus and nox (SciTools#4152) update bug and feature templates (SciTools#4147) Update CI environment lockfiles + Cartopy 0.19 changes (SciTools#4125) separate arg types from descriptions (SciTools#4100) Use assertArrayAllClose for sqrt test (SciTools#4118) Removed branch suffix (SciTools#4117) Corrected plot_anomaly_log_colouring for new Matplotlib linscale rules. (SciTools#4115) replace most recent hashes (SciTools#4112) Linkcheck update (SciTools#4104) CI lockfiles (SciTools#4108) Fix bug in gallery_tests runner (SciTools#4111) add logo to conda and pypi badges (SciTools#4088) pre-commit-ci update (SciTools#4092) Fix cirrus ci and mpl (SciTools#4087) Update readme docs stable (SciTools#4089) linux kernel version fix for conda 4.10+ (SciTools#4084) update docs pypi instructions (SciTools#4077) update flake8 pre-commit (SciTools#4067) Add GitHub discussions badge (SciTools#4070) ...

jamesp and others added 14 commits April 27, 2021 17:59

introducing lock files for testing

a9a4757

Testing github refresh workflow

642a03e

Update refresh-lockfiles.yml

bb444a8

Update refresh-lockfiles.yml

bd58e70

Action making lockfiles and setting as artifacts

f6c2ca5

Update refresh-lockfiles.yml

596d2a0

Update refresh-lockfiles.yml

9dea68d

Update refresh-lockfiles.yml

81ba3aa

Update refresh-lockfiles.yml

26b3e35

Update refresh-lockfiles.yml

5552016

Added PR creation

ddb3a8e

Cleanup refresh-lockfiles.yml

b20615c

Comments and cron scheduling for refresh-lockfiles workflow

Update .cirrus.yml

8f71bc1

noxfile now handles out of date environments, so no need to invalidate the cache in the cirrus configuration

Update refresh-lockfiles.yml

fc07f17

jamesp requested a review from bjlittle April 27, 2021 18:44

jamesp changed the title ~~Ci lock~~ CI lockfiles Apr 27, 2021

jamesp mentioned this pull request Apr 28, 2021

Richer Unstructured Plotting Support #4109

Closed

bjlittle self-assigned this Apr 28, 2021

bjlittle added Experience: High New: Feature Type: Enhancement Type: Infrastructure labels Apr 28, 2021

trexfeathers mentioned this pull request Apr 30, 2021

Update hashes for gallery wind plot #4112

Merged

rcomer mentioned this pull request May 3, 2021

Use numpy.linspace to generate grids built with reference point and spacings #4113

Closed

jamesp added 3 commits May 4, 2021 10:33

Updated imagehashes

a539b25

Picked from SciTools@566f187

Updated documentation of the github workflow

5b3e876

Merge branch 'ci-lock' of github.com:jamesp/iris into ci-lock

e3af577

* 'ci-lock' of github.com:jamesp/iris: Apply suggestions from @trexfeathers code review

rcomer linked an issue May 4, 2021 that may be closed by this pull request

Gallery: wind plot missing lake features #4110

Closed

Updated image hash for anomaly log plot

3301584

jamesp mentioned this pull request May 4, 2021

Added new image for anomaly plot in matplotlib > 3.4 SciTools/test-iris-imagehash#43

Closed

bjlittle assigned trexfeathers and unassigned bjlittle May 4, 2021

bjlittle removed their request for review May 4, 2021 13:49

Reverting imagerepo.json to state at 48e39b5

5241637

trexfeathers approved these changes May 4, 2021

View reviewed changes

trexfeathers merged commit f5ceb80 into SciTools:master May 4, 2021

jamesp mentioned this pull request May 12, 2021

Dependencies update #4130

Merged

trexfeathers mentioned this pull request May 18, 2021

Conda lock SciTools/iris-esmf-regrid#68

Merged

valeriupredoi mentioned this pull request May 19, 2021

Use a conda lock to create the env only once a week ESMValGroup/ESMValCore#1139

Closed

rcomer mentioned this pull request Jul 7, 2021

Checkout CI testing. #4232

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI lockfiles #4108

CI lockfiles #4108

jamesp commented Apr 27, 2021 •

edited

Loading

jamesp commented Apr 27, 2021

bjlittle commented Apr 28, 2021

bjlittle commented Apr 28, 2021

jamesp commented Apr 28, 2021

trexfeathers commented Apr 28, 2021

jamesp commented Apr 28, 2021

bjlittle commented Apr 28, 2021

bjlittle commented Apr 28, 2021 •

edited

Loading

rcomer commented Apr 29, 2021 •

edited

Loading

jamesp commented Apr 29, 2021

rcomer commented Apr 29, 2021

jamesp commented Apr 29, 2021

jamesp commented May 4, 2021

trexfeathers commented May 4, 2021 •

edited

Loading

trexfeathers commented May 4, 2021

jamesp commented May 4, 2021

rcomer commented May 4, 2021

jamesp commented May 4, 2021

trexfeathers left a comment

CI lockfiles #4108

CI lockfiles #4108

Conversation

jamesp commented Apr 27, 2021 • edited Loading

🚀 Pull Request

Description

jamesp commented Apr 27, 2021

bjlittle commented Apr 28, 2021

bjlittle commented Apr 28, 2021

jamesp commented Apr 28, 2021

trexfeathers commented Apr 28, 2021

jamesp commented Apr 28, 2021

bjlittle commented Apr 28, 2021

bjlittle commented Apr 28, 2021 • edited Loading

rcomer commented Apr 29, 2021 • edited Loading

jamesp commented Apr 29, 2021

rcomer commented Apr 29, 2021

jamesp commented Apr 29, 2021

jamesp commented May 4, 2021

trexfeathers commented May 4, 2021 • edited Loading

trexfeathers commented May 4, 2021

jamesp commented May 4, 2021

rcomer commented May 4, 2021

jamesp commented May 4, 2021

trexfeathers left a comment

Choose a reason for hiding this comment

jamesp commented Apr 27, 2021 •

edited

Loading

bjlittle commented Apr 28, 2021 •

edited

Loading

rcomer commented Apr 29, 2021 •

edited

Loading

trexfeathers commented May 4, 2021 •

edited

Loading