Computing bias/rmse/leaderboard from OutputVars

[Sbozzolo]

⚠️ This SDI is actively being written

The Climate Modeling Alliance

Software Design Issue 📜

Purpose

ClimaCoupler implements a "Leaderboard" module. Fundamentally, this module does two things:

1. Allow users to specify observational data and compute bias and RMSE between simulated data and observations
2. Allow users to specify RMSEs for other models and compare how our model performs

While the module works, there are some drawbacks, mostly stemming from the fact that the code is embedded in ClimaCoupler

• it clunky to use if one has already some output
• small modifications require going through the entire ClimaCoupler CI (which takes >1 h)
• it cannot be used by other packages

With this SDI, we move the module from ClimaCoupler to ClimaAnalysis. In the process, we update it to make use of newer features recently introduced in ClimaAnalysis (that will lead to simpler code).

Cost/Benefits/Risks

Benefits include:

• Fixing the problems above
• Allowing ClimaLand to directly use this code for comparing against observations
• Extracting some code that is useful in itself (e.g., computing integrals of OutputVars)

The logic of the code is already implemented and working, so the main cost is developer time.

People and Personnel

• Lead: @ph-kev
• Collaborators: @Sbozzolo
• Reviewers: @Sbozzolo

Components

Inputs

Results and Deliverables

Scope of Work

At higher level, there are two main components:

1. computing/plotting bias/rmse
2. computing/plotting leaderboard

Bias/RMSE

Beta Give feedback

1. Port split_by_season #59
   +0
   
2. Port constrained_cmap #62
   +0
3. If needed, extend OutputVar reader to support more generic NetCDF files #68
   +0
4. Use new interpolators to add resampled_as, a function that resamples an OutputVar to the same dimensions as another one #65
   +0
   
5. Port integrate_* functions to allow for integrating of an OutputVar. In this, we follow the same pattern as the average_* functions. #75
   +0
   
6. Add functions to compute error between sim and obs #84
   +0
   
7. Port plot_bias #86
   +0
   
8. Add split_by_season for OutputVar #78
   +0
   
9. Provide functions for unit conversions
10. Add struct and constructor for storing RMSEs across model names and seasons #88
    +0
    
11. Add find_best_single_model, find_worst_single_model, and median for struct that store RMSEs #89
    +0
    
12. Add plot_box_plot and plot_leaderboard using structs that store RMSEs #90
    +0
    
13. Add reordered_as, a function that reorders an OutputVar to match the ordering of dimensions of another one #94
    +0
    
14. Add extrapolation to constructors for OutputVar #95
    +0
    
15. Fix formatting and display of legend in box plot #97
    +0
    
16. Fix issue with interpolation of dates #99
    +0
    
17. Add support for converting between dates to seconds for time/date dimension #98
    +0
    
18. Refactor leaderboard code in ClimaCoupler.jl
19. Add leaderboard in ClimaLand.jl

Once this is completed, we will be able to directly compute bias and mse from two OutputVars. Packages like ClimaCoupler will load simulation data and observation data as OutputVars.

SDI Revision Log