Do not realize data and use dask in derivation functions

[mattiarighi]

As reported by @bouweandela, many derivation functions still realize the data and use numpy instead of dask. This is detrimental for the performance and should be changed.

Affected variables:

• amoc
• gtfgco2 should not be needed anymore, there is a preprocessor function for this
• sm
• toz

[bouweandela]

introduction to lazy data: https://scitools.org.uk/iris/docs/latest/userguide/real_and_lazy_data.html

If you need array functions to do things, use from dask import array as da instead of import numpy as np, see here for a description of dask array options: http://docs.dask.org/en/latest/array-api.html

[bouweandela]

See here for an example on multiplying a cube with a number without realizing data:
https://github.com/ESMValGroup/ESMValTool/blob/822941f52780dde2b0b122a9a8a99f23e313ef30/esmvaltool/cmor/_fixes/CMIP5/BNU_ESM.py#L154

And here for an example on using dask arrays:
https://github.com/ESMValGroup/ESMValTool/blob/822941f52780dde2b0b122a9a8a99f23e313ef30/esmvaltool/cmor/_fixes/CMIP5/BNU_ESM.py#L177-L178

[valeriupredoi]

just want to say, I ❤️ this thread 😁

[mattiarighi]

Great! Then it's yours 👍

[valeriupredoi]

yay, more crap for me!
FYI to us all, at the meeting with the iris folk last week I asked if they could explicitly say which iris funcs realize or keep the data lazy and Corinne has already started working on this (very important) info: SciTools/iris#3292

[valeriupredoi]

As reported by @bouweandela, many derivation functions still realize the data and use numpy instead of dask. This is detrimental for the performance and should be changed.

Affected variables:

* [] `amoc`

• this should be fine, no actual accessing of the data member

* [] `gtfgco2`

• this one needs to: remove the data access and mask construction and remove the building of a list of numpy arrays (bad worlf!)

* [] `sm`

• just the last bit that builds the mask from the data

* [] `toz`

• total mess, how do you set dtype to a dask array?

[valeriupredoi]

@zklaus would you be too angry with me if I asked you to look at this? I have a metric ton of crap that I need to take care of and I feel I am going to sideline this - plus you are working closely with the iris stuff anyways. Beer from me when we next meet 🍺

[ledm]

Similar to this discussion, is there a way to switch off writing the derived variables to disk? It seems to slow everything down and shouldn't be necessary.

[bouweandela]

You probably mean the input variables needed to derive a variable? In that case the answer is no.

[ledm]

Is it not possible to load the cubes into dask arrays, instead of saving them?

In the case of the derivation of OHC, it loads a 4D variable and saves it exactly as it is. It basically copies 20GB of data into the working directory for each dataset before doing any calculations! All I want is a scalar field, it should be a few kb!

Furthermore, we only have 100GB space in our home directories on jasmin, this means that there's only space for a few models using this method. (I will move my working directory somewhere with more space, but this still doesn't seem like a great method!)

[bouweandela]

Is it not possible to load the cubes into dask arrays, instead of saving them?

Maybe in the future, but not at the moment. Do you feel like implementing this yourself?

Furthermore, we only have 100GB space in our home directories on jasmin, this means that there's only space for a few models using this method. (I will move my working directory somewhere with more space, but this still doesn't seem like a great method!)

The Jasmin user guide recommends using a group workspace for storing large amounts of data: https://help.jasmin.ac.uk/article/176-storage, not your home directory. I started on pull request #265. which will make it possible to store preprocessor and other temporary data on a special temporary file system, but this is not ready yet.

[ledm]

Just a comment that gtfgco2 may still be needed. I've commented on the merged PR here #418 (comment)

but happy to continue the discussion here if needed.

[bouweandela]

Up-to-date overview and discussion in #2451.