Apply recipe(s) on a collection of datasets (recipe template, recipe filler...)

[mpmoine]

Dear ESMValtool Team,

We are curently thinking about a solution to apply a recipe (or a few selected recipes) on a collection of input datasets we have locally and this in an automatic way.

What we plan is to have a template of the recipe(s) and a script to fill it(them). We also have in mind to change the output_dir in the user_config file depending on the input dataset (model, experiment, member(s)...) so as to cleary distinguish the results from a same recipe (a tree structure similar to the DRS) applied on different datasets. The final objective is to make the resulting plots availaible in an per dataset (let's say a simulation) html page organised following this tree directory structure.

Searching in the ESMValCore/Tool documentation, we found (in the "Utils" section) some interesting functions like recipe_filler, test_recipe_filler.py (including a write_config_user function) and summary.py (usefull to generate the html page gathering all recipes results).

In January, during the IS-ENES3 final meeting in Toulouse, I discussed briefly with @remi-kazeroni and @sloosvel about what is in the BSC workfow. Even if the idea here is not to integrate ESMValTool in our modelling workflow at runtime (at least not yet 😉), I think the principle of template recipes, a filler and of the systematic ESMValTool execution we want to do offline has some similarities.

The point of this discussion is to get your feedbacks on the method we plan to apply here, what you think about reusing the above mentionned ESMValTool utilities. Do you think these functions are easily reusable by ESMValTool users (who are not developers)? Or may be simpler to write our own scripts? If you have any showable/exportable examples of how you proceed at BSC or elsewhere, we are interested!

Many thanks in advance,
Marie-Pierre and Hannah 🙂

[mpmoine]

From the 2.8.0 release page, I've just found the Automatically populating a recipe and notebook example for composing a recipe

it sounds that upgrading from our current 2.7.0 to 2.8.0 is what we have to do! 👍

Marie-Pierre.

[remi-kazeroni]

Hi @mpmoine, I don't have much experience myself with this topic but I would strongly recommend to make a new ESMValTool installation (in parallel to v2.7.0) to benefit from the latest capabilities you're mentioning. I think these newly added data finding and automatic recipe composition primarily target ESGF data but that should also work with local data if your native model is supported in ESMValCore (see here). Integrating ESMValTool to model development workflows has received and will continue to receive a lot of attention. Great to read that you are interested in taking this route 😃

[ehogan]

Hi @mpmoine 👋 I wanted to let you know that we (at the Met Office) are in the process of developing a Cylc workflow to automatically run ESMValTool recipes (also off-line, for now!) on our local model outputs for evaluation purposes. We are intending to automatically adapt / update already existing recipes in ESMValTool to enable our local model outputs to be used, but we haven't yet reached this point in our development, so we'd be keen to hear how you get on! Happy to talk more about this! :)

[mpmoine]

Hi @ehogan 👋 Glad to read that news from your Met Office side! Here at Meteo-France/Cerfacs we are have not yet adopted the Cylc task manager, neither for our modelling workflow nore for our offline data management/analysis part, but plan to test it (at least for this latter usage). And for the automatic recipe adaptating/updating, yes, we'll let you know where we are at the end of Hannah's internship (end of June).
Let's keep in touch! 🍻