Using preprocessors on unstructured data

[schlunma]

In this discussion I want to collect information about how our preprocessors perform on unstructured data. This is motivated by the fact that many of them currently don't work on this kind of data.

Below I will collect a quick overview. Each preprocessor that has been tested is added with a box, the box is ticked if it works directly on the unstructured data (without prior regridding!). For preprocessors that do not work, a new discussion item can be opened to discuss progress.

Fell free to add new preprocessors if you discover that they don't work! Thanks!

• area_statistics (fails with ValueError: Invalid operation for 'latitude', with 3 bound(s))
• climate_statistics
• extract_levels
• extract_region (fails with ValueError: expected 0 or 2 bound values per cell)
• regrid (when using appropriate schemes like unstructured_nearest)
• zonal_statistics (fails with ValueError: Invalid operation for 'latitude', with 3 bound(s))

[schlunma]

area_statistics and zonal_statistics

Fails with

ValueError: Invalid operation for 'latitude', with 3 bound(s). Contiguous bounds are only defined for 1D coordinates with 2 bounds.

Debug log

main_log_debug_area.txt
main_log_debug_zonal.txt

Problem description

This is an issue of iris that is already reported here: SciTools/iris#4672

Fix

This needs to be fixed in iris. I will open a PR as soon as I have the confirmation that the proposed solution in SciTools/iris#4672 is appropriate.

[schlunma]

extract_region

Fails with

ValueError: expected 0 or 2 bound values per cell

Debug log

main_log_debug.txt

Problem description

iris.cube.Cube.intersection used here cannot handle unstructured grids.

Fix

This could be solved by using iris.cube.Cube.extract with an appropriate constraint like

iris.Constraint(latitude=lambda cell: start_latitude < cell.point < end_latitude) &
iris.Constraint(longitude=lambda cell: start_longitude < cell.point < end_longitude)

which would most likely also work for regular grids.

Is there any particular reason we use Cube.intersection here? Does anyone know @ESMValGroup/esmvaltool-coreteam ?

[schlunma]

Ah, I didn't. In this case, extract either gives only the values between 0 and 10 (for start_longitude: -10, end_longitude: 10) or no cube at all (for start_longitude: 10, end_longitude: -10) as expected for a longitude that uses values between 0 and 360.

Our current preprocessor gives a cube with lons in [0, 10] and [350, 360] for the first case and an error (ValueError: minimum greater than maximum) for the second case.

Thus, intersect is a bit smarter than extract in terms of circular coordinates.

[bouweandela]

I think it would be best to open an issue in the iris repo about this and ask: why are there two ways to extract a region and why does it not work for all grids/regions?

[zklaus]

There are two things that set intersection apart from extract:

• extract works with arbitrary Constraints, intersection specifically with coordinate ranges
• intersection rolls circular coordinates, extract simply extracts

I think it's reasonably clear 🤷

[schlunma]

@zklaus That kind of makes sense, it's only weird that it also works for latitude and not only for longitude.

In this case, it would be great if you could upvote the issue referenced by Bouwe (SciTools/iris#4197) with a 👍.

[zklaus]

Done.