Profiling the VIC Image / CESM Drivers

[jhamman]

There has been recent discussion on developing profiling utilities for VIC, particularly the image and CESM drivers. I've broke the profiling down into 3 main categories based on what we discussed.

Timing

• Clock time for each grid cell for each timestep, basically time spent in vic_run. This would allow us to better understand what types of grid cells / physical situations cause the model to slow down. Done in Profiling: Feature/timing #520.
• Driver level timing table so we know where time is being spent (e.g. vic_force, put_data, vic_run). This would allow us to do a rudimentary timing table similar to what is done in CESM. Done in Profiling: Feature/timing #520.
• Off line timing comparisons of test datasets under a range of model settings. The current documentation has something along these lines for a very old (4.0.0) version of the model (ref).

Memory

• Along with no. 3 in the timing section, we should also profile the memory requirements. There are offline memory profilers that are available to do this (e.g. memprof, valgrind-massif, or tcmalloc).

Scaling

• How does the VIC MPI application scale as we add more processors? How much time is spent on I/O vs. computation vs. wait time from other processes? These questions can be answered with existing profiling tools that ship with mpich and openmpi.

Profiling

• Gprof call graph profiling.

So timing 1 and 2 are proposed to be added to the VIC source code while timing 3 and the memory and scaling profiles could be accomplished with little to no changes to the VIC source code.