GPM IMERG: benchmark virtual and zarr icechunk datasets for comparison with each other and current GES DISC time series service performance

[abarciauskas-bgse]

Christine Smit at GES DISC shared that their time series service can pull out 15 years of half-hourly data in about 11-12 seconds, about 260,000 time points. She also shared a notebook and locust file for testing the service's performance.

This issue is to capture that we want to provide similar benchmarks for the GPM IMERG virtual and zarr datasets.

As noted #1, the currently virtual implementation may be quite slow to open and read the data back out due to the size of the chunk manifests. If possible, I think it would be good to use this type of benchmark for each stage of development - so we can test the performance as things currently stand and also if improvements are made.

[rabernat]

Please share more details about the underlying data (chunks, etc.)