Lazy mask_fillvalues preprocessor function

[bouweandela]

Description

Make the mask_fillvalues preprocessor function lazy.

Note that this changes how provenance is recorded: now all cubes/datasets that are used as input to the function are considered ancestors of the output cubes/datasets. Previously, input cubes/datasets were only considered ancestors if they contributed to the shared mask. However, for lazy computations, this requires loading all the input data from disk and doing the computations before mask_fillvalues twice: once for computing the provenance and once for computing the result. This seemed too much of a computational burden for a feature that is very little used and therefore the choice was made to have less fine-grained provenance.

Needs Iris v3.9 (SciTools/iris#5775)

---

Before you get started

• ☝ Create an issue to discuss what you are going to do

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

• 🧪 The new functionality is relevant and scientifically sound
• 🛠 This pull request has a descriptive title and labels
• 🛠 Code is written according to the code quality guidelines
• 🧪 and 🛠 Documentation is available
• 🛠 Unit tests have been added
• 🛠 Changes are backward compatible
• 🛠 The list of authors is up to date
• 🛠 All checks below this pull request were successful

---

To help with the number pull requests:

• 🙏 We kindly ask you to review two other open pull requests in this repository

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 94.61%. Comparing base (d37a502) to head (119fefb).
Report is 34 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2340      +/-   ##
==========================================
+ Coverage   94.52%   94.61%   +0.09%     
==========================================
  Files         246      246              
  Lines       14036    14032       -4     
==========================================
+ Hits        13267    13276       +9     
+ Misses        769      756      -13     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[bouweandela]

The tests pass, but the tests are not very comprehensive. Comparing data coming out diagnostic_1/ta of examples/recipe_preprocessor_test.yml with the comparison tool suggests results are different.

[bouweandela]

Comparing data coming out diagnostic_1/ta of examples/recipe_preprocessor_test.yml with the comparison tool suggests results are different.

This has now been solved: da.ones_like(a) with a a masked dask array drops the mask, while np.ones_like(a) with a a masked numpy array keeps the mask.

[schlunma]

Thanks Bouwe, looks very good to me already! Just have some minor comments.

[schlunma]

Thanks Bouwe! Will approve once the merge conflict is solved and all tests are green 🚀

[bouweandela]

Thanks for reviewing @schlunma! Should be good to go now.

[schlunma]

🚀

[valeriupredoi]

great work folks - @bouweandela you want this in the 2.11 release or you think it can wait for later on after the release? If it can wait, pls add M2.12 to it 🍺

[bouweandela]

Thanks V! This is for the v2.12 release.

[valeriupredoi]

good man 🍺