native6: storing cmorized data generalizing the script cmorizers/era5.py

[bsolino]

First of all, a disclaimer:
I am not sure where on the several available places should this topic be discussed. I chose this forum because:

1. the script in question is on the ESMValTool project folder
2. I view this discussion as more about if it makes sense to do something, instead of how to do it
3. the discussions feature is new and I wanted to test it

So now back to the topic itself.

I am currently working on an EMAC on-the-fly cmorizer. The ERA5 is also available through an on-the-fly cmorizer, and there is a script that allows to take that data and store it to disk, which basically works as a normal cmorizer.

If my understanding of the process is correct, this script doesn't have to restrict itself to ERA5 data. What this script is doing is taking variables that were selected in the recipe and storing them to disk. All the information necessary for this process should be in memory and it has already be standardized, so I believe this could be a generic "store to disk" script that just stored the data according to the current standards.

However, I am still inexperienced so I may be missing some ERA5 specific information that is present on the script that I am unaware of, or any technical complication that necessitates to have independent scripts depending on the type of data you want to store.

So, what are your thought on this? Would it be possible? Is this something we want to do?

[bouweandela]

Actually the script renames the preprocessed files so they follow the default OBS6 naming convention as defined here: https://github.com/ESMValGroup/ESMValCore/blob/034f4e4f0ff367f8946aae04a7a69f5946bb02dc/esmvalcore/config-developer.yml#L186
so indeed it is fairly generic and could be used for other datasets too.

I would like to stress though, that in most cases there should be no need to store the CMORized data, because the CMORization process itself introduces very little computational overhead.

Specifically in the case of ERA5, we are interested in using daily data that is computed from hourly data. Because this provides a factor of 24 reduction in the data size and does take some time to compute, it is useful to store the CMORized daily data and therefore we created this recipe and script. For ERA5 data at hourly and monthly frequencies it probably doesn't make sense to store the CMORized data, that would just be duplicating the dataset, which is quite large already for hourly frequency.

Some previous discussion on this topic can be found in #1120 and in #1909.