Made ICON output UGRID-compliant (on-the-fly)

[schlunma]

Description

This is a first suggestions of an on-the-fly ICON -> UGRID converter for the ICON CMORizer, which eventually will allow us to use more sophisticated regridding algorithms (e.g., from iris-esmf-regrid) on ICON data.

I successfully tested this using the area weighted scheme from iris-esmf-regrid (I had to make some other modifications to the code, though; see below), which gave very similar results to CDO's remapcon:

Difference between CDO and iris-esmf-regrid for tas

> cdo diffn icon_cdo_remapcon.nc icon_esmvaltool_area_weighted.nc
               Date     Time   Level Gridsize    Miss    Diff : S Z  Max_Absdiff Max_Reldiff : Parameter name
     1 : 1979-01-31 23:45:00       2    64800       0   54086 : F F     0.016022  6.0465e-05 : tas        
     2 : 1979-02-28 23:45:00       2    64800       0   54008 : F F     0.016846  6.3353e-05 : tas        
     3 : 1979-03-31 23:45:00       2    64800       0   54156 : F F     0.016174  5.8117e-05 : tas
     4 : 1979-04-30 23:45:00       2    64800       0   53986 : F F     0.017212  6.2066e-05 : tas        
     5 : 1979-05-31 23:45:00       2    64800       0   54004 : F F     0.016296  5.7541e-05 : tas        
     6 : 1979-06-30 23:45:00       2    64800       0   52898 : F F     0.015015  5.1399e-05 : tas        
     7 : 1979-07-31 23:45:00       2    64800       0   52760 : F F     0.015991  5.4423e-05 : tas        
     8 : 1979-08-31 23:45:00       2    64800       0   53075 : F F     0.014465  4.9827e-05 : tas        
     9 : 1979-09-30 23:45:00       2    64800       0   53385 : F F     0.014343  5.0131e-05 : tas        
    10 : 1979-10-31 23:45:00       2    64800       0   53781 : F F     0.013397  4.7200e-05 : tas        
    11 : 1979-11-30 23:45:00       2    64800       0   53982 : F F     0.016266  5.9785e-05 : tas        
    12 : 1979-12-31 23:45:00       2    64800       0   53959 : F F     0.016815  6.1995e-05 : tas        
    13 : 1980-01-31 23:45:00       2    64800       0   54116 : F F     0.018555  6.8959e-05 : tas        
    14 : 1980-02-29 23:45:00       2    64800       0   54244 : F F     0.017151  6.2585e-05 : tas        
    15 : 1980-03-31 23:45:00       2    64800       0   54316 : F F     0.016602  5.9685e-05 : tas        
    16 : 1980-04-30 23:45:00       2    64800       0   54169 : F F     0.017548  6.1626e-05 : tas        
    17 : 1980-05-31 23:45:00       2    64800       0   53795 : F F     0.017029  5.8690e-05 : tas        
    18 : 1980-06-30 23:45:00       2    64800       0   52975 : F F     0.014923  5.1183e-05 : tas        
    19 : 1980-07-31 23:45:00       2    64800       0   52917 : F F     0.017029  5.7629e-05 : tas        
    20 : 1980-08-31 23:45:00       2    64800       0   53377 : F F     0.015289  5.2392e-05 : tas        
    21 : 1980-09-30 23:45:00       2    64800       0   53694 : F F     0.015533  5.4426e-05 : tas        
    22 : 1980-10-31 23:45:00       2    64800       0   53768 : F F     0.013153  4.9979e-05 : tas        
    23 : 1980-11-30 23:45:00       2    64800       0   54043 : F F     0.015015  5.5051e-05 : tas        
    24 : 1980-12-31 23:45:00       2    64800       0   54219 : F F     0.016510  6.1575e-05 : tas        
    25 : 1981-01-31 23:45:00       2    64800       0   54068 : F F     0.016693  6.2804e-05 : tas        
    26 : 1981-02-28 23:45:00       2    64800       0   53982 : F F     0.018280  6.8291e-05 : tas        
    27 : 1981-03-31 23:45:00       2    64800       0   53818 : F F     0.016754  6.0377e-05 : tas        
    28 : 1981-04-30 23:45:00       2    64800       0   53885 : F F     0.017029  6.0841e-05 : tas        
    29 : 1981-05-31 23:45:00       2    64800       0   53411 : F F     0.015045  5.1801e-05 : tas        
    30 : 1981-06-30 23:45:00       2    64800       0   52816 : F F     0.015564  5.3491e-05 : tas        
    31 : 1981-07-31 23:45:00       2    64800       0   52711 : F F     0.014740  4.9659e-05 : tas        
    32 : 1981-08-31 23:45:00       2    64800       0   53669 : F F     0.015228  5.1716e-05 : tas        
    33 : 1981-09-30 23:45:00       2    64800       0   53383 : F F     0.014984  5.2563e-05 : tas        
    34 : 1981-10-31 23:45:00       2    64800       0   53928 : F F     0.013916  5.0085e-05 : tas        
    35 : 1981-11-30 23:45:00       2    64800       0   54461 : F F     0.012390  4.5595e-05 : tas        
    36 : 1981-12-31 23:45:00       2    64800       0   54254 : F F     0.015442  6.0295e-05 : tas        
    37 : 1982-01-31 23:45:00       2    64800       0   54353 : F F     0.017700  6.6394e-05 : tas        
    38 : 1982-02-28 23:45:00       2    64800       0   54144 : F F     0.016510  6.1102e-05 : tas        
    39 : 1982-03-31 23:45:00       2    64800       0   54165 : F F     0.017120  6.2717e-05 : tas        
    40 : 1982-04-30 23:45:00       2    64800       0   54213 : F F     0.017059  6.0221e-05 : tas        
    41 : 1982-05-31 23:45:00       2    64800       0   53907 : F F     0.015717  5.4128e-05 : tas        
    42 : 1982-06-30 23:45:00       2    64800       0   53169 : F F     0.016144  5.5411e-05 : tas        
    43 : 1982-07-31 23:45:00       2    64800       0   52942 : F F     0.017365  5.8832e-05 : tas        
    44 : 1982-08-31 23:45:00       2    64800       0   52971 : F F     0.015961  5.4694e-05 : tas        
    45 : 1982-09-30 23:45:00       2    64800       0   53607 : F F     0.015289  5.3670e-05 : tas        
    46 : 1982-10-31 23:45:00       2    64800       0   53790 : F F     0.014893  5.3966e-05 : tas        
    47 : 1982-11-30 23:45:00       2    64800       0   53991 : F F     0.016846  6.2195e-05 : tas        
    48 : 1982-12-31 23:45:00       2    64800       0   54553 : F F     0.016876  6.3282e-05 : tas        
    49 : 1983-01-31 23:45:00       2    64800       0   54450 : F F     0.017761  6.5861e-05 : tas        
    50 : 1983-02-28 23:45:00       2    64800       0   54588 : F F     0.018524  6.8562e-05 : tas        
    51 : 1983-03-31 23:45:00       2    64800       0   54384 : F F     0.015991  5.8340e-05 : tas        
    52 : 1983-04-30 23:45:00       2    64800       0   54051 : F F     0.017273  6.1452e-05 : tas        
    53 : 1983-05-31 23:45:00       2    64800       0   53856 : F F     0.017456  6.0236e-05 : tas        
    54 : 1983-06-30 23:45:00       2    64800       0   53162 : F F     0.015167  5.2268e-05 : tas        
    55 : 1983-07-31 23:45:00       2    64800       0   52911 : F F     0.015076  5.1411e-05 : tas        
    56 : 1983-08-31 23:45:00       2    64800       0   53230 : F F     0.015289  5.2364e-05 : tas        
    57 : 1983-09-30 23:45:00       2    64800       0   53904 : F F     0.013611  4.7854e-05 : tas        
    58 : 1983-10-31 23:45:00       2    64800       0   53947 : F F     0.014526  5.2575e-05 : tas        
    59 : 1983-11-30 23:45:00       2    64800       0   54294 : F F     0.012909  5.0397e-05 : tas        
    60 : 1983-12-31 23:45:00       2    64800       0   54500 : F F     0.015335  6.1684e-05 : tas

Closes #1653

Link to documentation: https://esmvaltool--1664.org.readthedocs.build/projects/ESMValCore/en/1664/quickstart/find_data.html#icon

Issues that need to be solved before merging this

• iris-esmf-regrid does not yet offer a iris.cube.Cube.regrid()-compatible scheme for their mesh regridder (I temporarily added this to test it) - see iris.cube.Cube.regrid()-compatible scheme for MeshToGridESMFRegridder SciTools/iris-esmf-regrid#192. --> not yet necessary anymore due to Add iris-esmf-regrid as a dependency #1809.
• The MeshCoord added to the cube using Mesh.to_MeshCoords always uses the metadata of the node coordinate and always uses var_name=None, which leads to errors in our CMOR checks, e.g., Coordinate longitude has var name None instead of lon - MeshCoord constructor always uses metadata of node coordinate regardless of location SciTools/iris#4860.

Related issues (would be nice to solve these before merging, but not crucial)

• To load the resulting cubes, the context manager iris.experimental.ugrid.PARSE_UGRID_ON_LOAD has to be used. Thus, diagnostics cannot read this data out-of-the box. I guess this will be removed by iris once meshes are fully supported and lose their "experimental" status.

After this PR

• Open issue about changing link to ICON grid file to GitHub once this file is on main and fix it (see Updated URL of ICON grid file used for testing #1914)

---

Before you get started

• ☝ Create an issue to discuss what you are going to do

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

• 🧪 The new functionality is relevant and scientifically sound
• 🛠 This pull request has a descriptive title and labels
• 🛠 Code is written according to the code quality guidelines
• 🧪 and 🛠 Documentation is available
• 🛠 Unit tests have been added
• Adapt unit tests once issues (see above) have been solved
• 🛠 Changes are backward compatible
• 🛠 Any changed dependencies have been added or removed correctly
• 🛠 The list of authors is up to date
• 🛠 All checks below this pull request were successful

---

To help with the number pull requests:

• 🙏 We kindly ask you to review two other open pull requests in this repository

[codecov]

Codecov Report

Merging #1664 (bc2cf5d) into main (ed3ae70) will increase coverage by 0.02%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main    #1664      +/-   ##
==========================================
+ Coverage   92.04%   92.07%   +0.02%     
==========================================
  Files         234      234              
  Lines       12057    12112      +55     
==========================================
+ Hits        11098    11152      +54     
- Misses        959      960       +1     

Impacted Files	Coverage Δ
esmvalcore/cmor/_fixes/icon/_base_fixes.py	98.21% <100.00%> (-1.79%)	⬇️
esmvalcore/cmor/_fixes/icon/icon.py	100.00% <100.00%> (ø)
esmvalcore/preprocessor/_regrid.py	96.58% <100.00%> (+0.01%)	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[pp-mo]

Regarding ...

To load the resulting cubes, the context manager iris.experimental.ugrid.PARSE_UGRID_ON_LOAD has to be used. Thus, diagnostics cannot read this data out-of-the box. I guess this will be removed by iris once meshes are fully supported and lose their "experimental" status.

You can in fact "turn on" the UGRID parsing permanently, by an alternate use of the context manager interface, albeit a bit clunky !

from iris.experimental.ugrid import PARSE_UGRID_ON_LOAD

# turn on permanently
permanent_load_context = PARSE_UGRID_ON_LOAD.context()
permanent_load_context.__enter__()

Note, however, that the context object here created must not go out of scope! (since if destroyed, it will turn off again)
So for example, PARSE_UGRID_ON_LOAD.context().__enter__() will not work.

A simpler but less clean way is just PARSE_UGRID_ON_LOAD._state = True, but obviously that is a bit naughty.

[valeriupredoi]

it would be quite nice if @bouweandela and @bsolino had a wee time look at this and review it. Cheers 🍺

[bouweandela]

I'll have a look as soon as possible, but last week I was caught up in the IS-ENES3 GA and this week I'm quite busy doing internal stuff for my employer. I will probably have some time later this week.

[bouweandela]

I reviewed the code and documentation and it looks good. I will try running some tests tomorrow.

[bouweandela]

I ran some tests with an adapted version of examples/recipe_python.yml, as shown below, and things seem to work fine, though I'm no expert on UGRIDs.

# ESMValTool
# recipe_python.yml
---
documentation:
  description: |
    Example recipe that plots a map and timeseries of temperature.

  title: Recipe that runs an example diagnostic written in Python.

  authors:
    - andela_bouwe
    - righi_mattia

  maintainer:
    - schlund_manuel

  references:
    - acknow_project

  projects:
    - esmval
    - c3s-magic

datasets:
  - {project: ICON, dataset: ICON, exp: cool014}


preprocessors:

  select_january:
    regrid:
      target_grid: 0.5x0.5
      scheme:
        reference: esmf_regrid.experimental.unstructured_scheme:regrid_unstructured_to_rectilinear
        method: bilinear
      lon_offset: false
    extract_month:
      month: 1

diagnostics:
  map:
    description: Global map of temperature in January 2000.
    themes:
      - phys
    realms:
      - atmos
    variables:
      tas:
        mip: Amon
        preprocessor: select_january
        start_year: 1980
        end_year: 1980
    scripts:
      script1:
        script: examples/diagnostic.py
        quickplot:
          plot_type: pcolormesh
          cmap: Reds

[bsolino]

I have read the code and all looks good :)

I've had a small issue when running the test that I'm not sure if it's on my end: It is looking for the ICON grid on my personal folder $HOME/.esmvaltool/cache/icon_grid_0043_R02B04_G.nc. How could I proceed?

[schlunma]

Thanks for reviewing @bouweandela and @bsolino!

After a quick offline discussino with @bsolino we found that the cache directory is not automatically created when it's not present, which leads to the error Brei mentioned. This is fixed now via a simple self.CACHE_DIR.mkdir(parents=True, exist_ok=True) 👍

[bsolino]

Everything looks good now, good work!