Performance optimization for reading input data

[samfredriksson]

Hi,
Quite new to opendrift – looking forward to get to know it. I´m wondering how to optimize performance regarding reading input data. I´m thinking of having either input data and/or time step at either 1 h or 10 min frequency. Having input data of 1h and time step in opendrift of 10 min is much faster than input data of 10 min and opendrift time step of 10 min which surprises me somewhat since the time interpolation is not needed for the same input frequency and opendrift time step. I reason this has to do with the reading of the input files since higher input frequency also leads to larger data set given the same analysis time. Is this the true reason or is it something else? Is there a way to come around this problem?

[knutfrode]

Hi Sam,

The reading of data from files (and in particular through Thredds/OPeNDAP) is much slower than the interpolation. So it makes sense that it is faster if data is read only each full hour.
OpenDrift uses internal caching, so that no data are read twice.
So best is probably to leave your input data at its original frequency. For internal time step, as fine as 10 minutes might be needed if you are close to a complex coastline (fjords, islands), otherwise 1 hour is often ok. Also note that the output time step (time_step_output) might be at a lower frequency than the calculation time step.

[samfredriksson]

Thanks for the fast reply
The coastline is indeed comlex in this application (with fjords and islands etc).
Can the access to the input files be tuned/controlled by some chunking, only some files at a time, more files with less time steps in each etc?

[knutfrode]

I am not sure what is meant here.
For this application, 10 minutes might seem like a suitable calculation time step.
But your forcing/input data presumably has lower temporal resolution than this? Then the best is just to use the raw forcing data as it is, and OpenDrift will automatically read only the minimum of data as needed, and never read the same data twice.
You may provide as many or few data sources as needed. OpenDrift will start from the first reader (by the order they are added), and will not use more readers as soon as necessary data are obtained (e.g. current and wind).
The output log (in debug mode) will show you which data has been read from which reader.

[samfredriksson]

I agree that 10 min is a suitable calculation time step. For the forcing data I´m presently in the process of deciding output frequency and am therefore doing a sensitivity study fo this purpose.
The forcing data can be outputted in daily, monthly etc files regardless of the time averaging frequency of the variable (as current speed). A monthly file therefore has some 30 times more time steps than a daily one. The file size incerases accordingly. Sometimes it´s more favourable from memory perspective to work with smaller files that are read for the time steps that are at hand. If all the files are read at the same time and then kept in the memory it does, however, not make any difference if you use a few large files or many small files.

I´m presently use something similar to

reader = reader_netCDF_CF_generic.Reader('nn_10mi_201609*_201609*_grid_UV.nc')

to read the U and V currents and * is the days of September.

[knutfrode]

OpenDrift is very economical regarding reading of data, as this is the bottleneck of most trajectory simulations.
For a given cloud of particles, a minimal subset (2D array, or 3D if vertical as well) that just covers the cloud is read for the closest timesteps before and after the present calculation time step. This is cached in memory, and interpolated in both time and space for coming time steps, until passing the "time after". Thereafter "time after" becomes "time before", and a new 2D/3D block of data is read that covers the present cloud. The old "time after" is then discarded from memory.
So the amount of data kept in memory at any time depends only on the geographical spread of your particles (independent of the number of particles). But with higher frequency of the forcing files, you will need to read data more often, but interpolation in time is better represented.

[samfredriksson]

I did a test on my one where I splitted the input (forcing) files into hourly instead of daily files, thereby decreasing the number of time steps in forcing each file from 144 to 6 and instead increasing the number of files by 24 to cover the same total running time. This made all the difference in our case (fairly large 3D-mesh) and we have now similar performance for either 1h or 10min forcing resolution.
Once again thanks for your answers