How should we manage non-reproducible recipes in the RTW?

[ehogan]

During the development of the Recipe Test Workflow (RTW; see ESMValTool: Issue: Recipe Test Workflow (RTW) prototype) we discovered that some recipes are not reproducible. We would like the community's input on how we should manage non-reproducible recipes in the RTW.

• Is making them reproducible worth saving the effort of checking them manually at release time?

Please add your thoughts and ideas here! 😊

[katjaweigel]

There are R diagnostics which dynamically set the colors (you get different ones every time), e.g. droughtindex/diag_spei.R from recipe_spei.yml. I assume that makes the recipe not reproducible and should be rather easy to change by setting fixed colors?