Replies: 2 comments
-
An alternate way would be that the same run directory can optionally be used again and again until successful run (thus avoiding to copy or link results from previous run between run directories) |
Beta Was this translation helpful? Give feedback.
-
@zklaus , @valeriupredoi , @bouweandela |
Beta Was this translation helpful? Give feedback.
-
Caching results could be much useful for compute efficiency. This has been experimented in CLiMAF. The attached document Caching_EVT_results.pdf explains how it is done in CLiMAF and drafts how it could be implemented in ESMValTool.
In email discussions :
@zklaus reported that :
while @valeriupredoi noted that :
@Bouwe added :
The reproducibility issue raises a question : what should be the scope of cache use ?
The actual experience with CliMAF is that the cache is mainly used on an individual basis, in the course of developing and refining an analysis, and not as a long-term archival of reproducible reference results. This means that cached results are useful on a timescale which is not prone to see much change of the dependencies, and that it could make sense to adopt such a limited scope for an ESMValTool cache. In such a case, this should of course be made very clear for users.
Additionally, reproducibility of ESMValTool results should rather rely on provenance information, which is best suited for hosting quite extensive dependencies description
The
resume
feature included from version 2.4 brings some progress for re-using previous results : it allows to re-use the output of the successful post-processing stages from a previous run, on a diagnostic+variable basis. Its limitations are, AFAIU :auto-resume
option, by which ESMValTool would re-use all run directories for the same recipe name in the sameoutput_dir
Dropping some of these limitations could be a good compromise for improving the situation
Beta Was this translation helpful? Give feedback.
All reactions