How to make it super-easy to use NWPs from multiple providers interchangeably?

[JackKelly]

One of the ultimate aims is to make it super-easy for users to add a new NWP to their product. Ideally, it'd be great if different NWPs could be as interchangeable as lego bricks.

In practice, this requires standardising:

1. Variable names
2. Dimension names
3. Physical units
4. Names of vertical levels; and possibly converting vertical levels to machine-readable form

Horizontal resolution & projection

NWPs from different providers also tend to have different horizontal resolutions and projections.

You could imagine a world in which these are reprojected on-the-fly to a "standard" resolution and projection, but that's computationally expensive, and introduces artefacts.

And, perhaps, it's not necessary (or desirable) to reproject. Attention-based ML models can handle different resolutions and projections (because we "manually" encode the location of each input, so all we have to do is tell the model the lat/long coords of each input pixel). And a lot of use-cases probably just want forecasts that are nearest to a target geospatial location (e.g. a particular wind farm).

Related

• Make standard and long names cf compliant mpiannucci/gribberish#50

[devsjc]

Definitely not an easy problem to solve! For what its worth, the decision I came to in my rewrite of the NWP consumer for OCF was to keep the grid as-provided, but standardize variable names, dimension names, and units, as that seemed to suit our ML pipelines the best.

[leonoverweel]

Variable names

I've found that this one is tricky because many NWPs output different variables. Even variables that sound like they should be the same across models will often differ in units or even what they're modeling exactly. ECMWF HRES has a pretty different DNI than ICON-EU for example.

Names of vertical levels

This has a similar issue as some models define them in meters and some define them in pressure levels, and they are all slightly different and you can't necessarily construct a one-to-one mapping between models.

---

What we've settled on at Dexter is not adding an abstraction layer like you suggest, but rather ensuring that all variables across NWPs are easily discoverable and queryable (using generalized client software and dashboards).

[JackKelly]

Sounds good! Thanks loads for the insight!