🧐 Subset WeatherBench2 to one year and select variables #3
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Obtaining a temporal subset of WeatherBench2 from 2020-2022, at pressure level 500hPa, with data variables: geopotential, u_component_of_wind, v_component_of_wind. Disabling compression when saving to Zarr so that it can be read back using cupy-xarray's kvikIO engine developed at xarray-contrib/cupy-xarray#10.
weiji14
changed the title
Changing from the 240x121 spatial resolution grid to 1464x721, so that a single time chunk is larger (about 4MB). Decreased time range from twenty years to one year (previous commit at c63bc43 was actual 20, not 2 years). Also had to rechunk along the latitude and longitude dims because the original WeatherBench2 Zarr store had 2 chunks per time slice.
weiji14
changed the title
weiji14
commented
Oct 5, 2023
| ds_500hpa_zuv[var].encoding["compressor"] = None | ||
|
|
||
| # Save to Zarr with chunks of size 1 along time dimension | ||
| # Can take about 1 hour to save 10.7GB of data at 40MB/s |
There was a problem hiding this comment.
Oops, forgot to update this line 😅
Benchmark download time at commit c63bc43 for 10.7GB of data was:
real 78m14.295s
user 25m23.062s
sys 5m0.360s
At commit a904131, dataset is 18.2GB and took:
real 793m42.489s
user 279m16.092s
sys 51m52.820s
Internet speeds on my side was averaging at around 40MB/s, and rechunking on the fly probably took some time 🙂
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Obtaining a temporal subset of WeatherBench2 for the year 2000 only, at pressure level 500hPa, with data variables: geopotential, u_component_of_wind, v_component_of_wind. Disabling compression when saving to Zarr so that it can be read back using cupy-xarray's kvikIO engine developed at xarray-contrib/cupy-xarray#10 (as of Oct 2023).
References: