Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
👥 Compare data loading between kvikIO and Zarr engine (#7)
* 👔 Get mean time by averaging over ten epochs Benchmark results between the Zarr and kvikIO engine were too close for one epoch, so looping over 10 epochs and reporting the average instead. Not printing the MSE Loss anymore to declutter the console output. * ➕ Add ipywidgets Jupyter Interactive Widgets! Repo at https://github.com/jupyter-widgets/ipywidgets * ♻️ Use tqdm.auto to also work in notebooks Will be reusing some of this code in a Jupyter Notebook, so refactoring to use tqdm.auto instead of standard tqdm. * 🔊 Report median time and standard deviation across epochs Save the time taken to complete each epoch, and compute the median, mean and standard deviation across all epochs. Needed because the time to process one epoch can vary by a few seconds across the ten epochs depending on various factors (e.g. caching), so computing the average time as total_time / num_epochs can lead to misleading results. Also updated main README.md to say be more specific about the reported total/median/mean/std benchmark times and the size of the ERA5 subset dataset. * 👥 Compare data loading between kvikIO and Zarr engine Reporting the actual numbers on which is faster - kvikIO or Zarr! Reusing some code from 1_benchmark_kvikIOzarr.py, but now the total/median/mean/std times can be displayed. Final cell calculates the speedup of kvikIO to be ~20% over Zarr, but note that this speedup can actually fluctuate depending on lots of factors (have seen values from 10%-30% over multiple runs). * ➕ Add seaborn Statistical data visualization in Python! * 💫 Plot bar graph showing time taken between kvikio and zarr engine A bar plot (with error bars) to visually compare kvikio (with GPUDirect Storage) against the zarr (no GPUDirect Storage) xarray backend engines in terms of data loading speed. Speedup results still fluctuates between runs, but are mostly around the 20% mark. Also did some slight refactoring to use pandas instead of numpy for the mean/median/std calculations. Using ddof=1 for the standard deviation. * 🐛 Change barplot estimator to median instead of mean Seaborn plots the mean value by default, but changing to median instead. The kvikIO engine is now reported as 35% faster than the Zarr engine. * 💬 Report as percentage less time, not speedup Speed is equal to Distance (or epochs) over time. It makes more sense to report 'less time' (absolute measure) instead of 'faster speed' (inverse measure), so fixing the formulation. Previous calculation of speedup may actually have been incorrect?
- Loading branch information