NWPs change shape over time. How to detect, store, and handle these changes?

[JackKelly]

xarray expects that datasets have the same shape and the same coordinate labels across the entirety of the dataset.

But NWPs get upgraded over time. For example, NWPs add more ensemble members, or increase resolution.

Detecting changes in the number of ensemble members, forecast steps, or vertical levels

These changes should be reflected in .idx files.

Detecting changes in horizontal resolution

Horizontal resolution isn't reflected in .idx files.

If the NWP has high-quality documentation then maybe the documentation will describe exactly when the horizontal spatial resolution changes, and we could read one GRIB file from each resolution change to get the coordinate labels.

Otherwise we'll have to do a binary search through the GRIB files to find each resolution change, and to get the coordinate labels. (Although, because cloud object storage only provides high bandwidth when many files are read in parallel, we might need an "over-eager binary search" which submits read requests before we know the result of all preceding checks).

Storing changes

AFAIK, no existing manifest format can express the concept of changing resolutions. So we'll have to come up with our own. Which shouldn't be too hard.

Handling changes in the number of ensemble members, forecast steps, or vertical levels

It might be sufficient to present the dataset as if it always had the highest number of ensemble members, forecast steps, and vertical levels. And just return NaN if the user requests older data that doesn't have all ensemble members etc.

For example, if an NWP has 25 ensemble members from 2015 to 2020, and 50 ensemble members from 2021 onwards, then the dataset's shape and coord labels would look as if it always had 50 ensemble members. But if the user asks for the 50th ensemble member from 2015 then they'll get NaNs.

Handling changes in horizontal resolution

The simplest approach is probably to treat each different-shaped NWP as a different xarray dataset.

In the future, it might be nice to offer the user the option of having the entire dataset presented to xarray as if the dataset were always at the highest resolution, and for the system to transparently up-sample lower-horizontal-resolution parts of the dataset.

Related

• Metadata that we'd like to store #17

[jacobbieker]

Would the goal of this to keep the data in the vertical resolution in pressure levels/meters above ground/etc? Or in the native vertical resolution (sigma/model levels)? When there is changes to the native vertical resolutions on the model level, I believe a lot of the time the same model level in the upgraded NWP does not correspond to the same model level in the pre-upgraded NWP.

[JackKelly]

Oooh, good point. The short answer is that I'm not sure! Except, of course, that the idea is that we're reading data directly from pre-existing GRIB files in cloud object storage. So we're constrained to using whatever coordinate labels exist in those GRIB files. I think?! Or have I missed the point of your question?!

[jacobbieker]

No, you got it! I guess, just something to think about if making a large, public manifest for an NWP, probably makes then the most sense to make it for the gribs that use physical vertical coordinates, rather than model levels, otherwise there might be some sneaky gotcha's for downstream users! E.g. for ICON, they have both the model level, and pressure level gribs located here https://opendata.dwd.de/weather/nwp/icon-eu/grib/09/t/ for a run.

[leonoverweel]

Since usually when the shape of an NWP changes, it coincides with the underlying model changing, I would just treat them as separate models and not try to combine them. Usually the model changes mean that there isn't much you can learn from the old, pre-(resolution-)change model that has a lot of value when using the new, post-(resolution-)change model as input day-to-day.