[recipe testing strategy] Testing recipes on a weekly basis

[remi-kazeroni]

Is your feature request related to a problem? Please describe.
During the meeting on the recipe testing strategy #2259 on October 14, 2021, we discussed the possibility to test some recipes more frequently than shortly before a release. These tests would be done on a weekly basis for a subset of recipes. This testing approach would help to check if changes in the Core affect the recipes. Several options were discussed to this end:

• In order to limit the computational efforts required, we would need to find a sensible way to split recipes in subsets. Would it make sense to run recipes that are known to fail or problematic more often than others? Shall the recipes with a too long runtime be excluded from this testing approach?
• What kind of computational resources would be needed for that? Are the resources available with the ESMValBot enough?
• Is it possible and relevant to perform this testing without calling the diagnostic scripts?
• How do we handle the "missing data problem"? And do we need to test data changes as well? What about the availability of observational data?
• Would it help to have a "test version" of each/many recipes that could be computationally less demanding, possibly using less datasets and a narrower time range? That would imply twice more recipes to maintain.
• Could we develop synthetic data on which all recipes would run? Consensus was reached that developing such dataset would be difficult as ESMValTool runs on a great variety of data. These data may also not capture the trends and thresholds analyzed in recipes.
• Could the recipes be tested using a very low resolution only? Not clear if that would work for all recipes.
• Should we define a better way to record backwards incompatible changes such as a "live changelogs"?

Feel free to edit this summary or to comment in this issue. (This issue could be transferred to the Core if more appropriate.)

[bouweandela]

Created ESMValGroup/ESMValCore#1636 to describe work done on using a special set of small recipes for testing ESMValCore.