ENSO on-the-fly plots

[lee1043]

Addresses #1127.

Accompanied PR in ENSO metrics: CLIVAR-PRP/ENSO_metrics#82

Demo notebook update to include sample dive down plots.

[lee1043]

At the moment, CI failing might be caused by constraint to enso metrics version as 1.1.3 while it is yet to release.

[lee1043]

@acordonez when you get a chance, could you please run demo 6 with the PR to see if it run smoothly?

[acordonez]

@lee1043 I'm having some issues creating an environment that works. It looks like cmocean need to be added to dev.yml? And I'm getting the following error:

Traceback (most recent call last):
  File "/home/ordonez4/miniconda3/envs/pmp_dev_enso/bin/enso_driver.py", line 24, in <module>
    from EnsoPlots.EnsoMetricPlot import main_plotter
  File "/home/ordonez4/miniconda3/envs/pmp_dev_enso/lib/python3.10/site-packages/EnsoPlots/__init__.py", line 1, in <module>
    from .EnsoMetricPlot import *
  File "/home/ordonez4/miniconda3/envs/pmp_dev_enso/lib/python3.10/site-packages/EnsoPlots/EnsoMetricPlot.py", line 8, in <module>
    from .EnsoPlotTemplate import cmip_boxplot, my_boxplot, my_curve, my_dotplot, my_dot_to_box, my_hovmoeller, my_map,\
  File "/home/ordonez4/miniconda3/envs/pmp_dev_enso/lib/python3.10/site-packages/EnsoPlots/EnsoPlotTemplate.py", line 12, in <module>
    from mpl_toolkits.basemap import Basemap
ModuleNotFoundError: No module named 'mpl_toolkits.basemap'

These are my matplotlib versions:

# Name                    Version                   Build  Channel
matplotlib                3.9.2           py310hff52083_2    conda-forge
matplotlib-base           3.9.2           py310h68603db_2    conda-forge
matplotlib-inline         0.1.7              pyhd8ed1ab_0    conda-forge

[lee1043]

@acordonez thank you for pointing that out which is a great reminder! I missed adding it to dependency. I will add them to dev and ci yml files.

[lee1043]

@acordonez just added. Could you check if that resolves the issue?

[acordonez]

@lee1043 That solved the cmocean import issue but I am still getting this error when running enso_driver.py:

Traceback (most recent call last):
  File "/home/ordonez4/miniconda3/envs/pmp_dev_enso2/bin/enso_driver.py", line 24, in <module>
    from EnsoPlots.EnsoMetricPlot import main_plotter
  File "/home/ordonez4/miniconda3/envs/pmp_dev_enso2/lib/python3.10/site-packages/EnsoPlots/__init__.py", line 1, in <module>
    from .EnsoMetricPlot import *
  File "/home/ordonez4/miniconda3/envs/pmp_dev_enso2/lib/python3.10/site-packages/EnsoPlots/EnsoMetricPlot.py", line 8, in <module>
    from .EnsoPlotTemplate import cmip_boxplot, my_boxplot, my_curve, my_dotplot, my_dot_to_box, my_hovmoeller, my_map,\
  File "/home/ordonez4/miniconda3/envs/pmp_dev_enso2/lib/python3.10/site-packages/EnsoPlots/EnsoPlotTemplate.py", line 12, in <module>
    from mpl_toolkits.basemap import Basemap
ModuleNotFoundError: No module named 'mpl_toolkits.basemap'

I tried using mamba install -c conda-forge basemap to install basemap explicitly but get this error:

Could not solve for environment specs
Encountered problems while solving:
  - package libgdal-core-3.10.0-hef9eae6_2 requires geos >=3.13.0,<3.13.1.0a0, but none of the providers can be installed

The environment can't be solved, aborting the operation

I can keep looking into this but curious if you have any ideas.

[lee1043]

@acordonez thank you for testing. It looks like I have those libraries in my environment, which I suspect they might be dependency of something. Can you check your maplotlib version? Mine is 3.9.2.

$ conda list libgdal

# Name                    Version                   Build  Channel
libgdal-core              3.10.0               h61e89c6_1    conda-forge

$ conda list geos

# Name                    Version                   Build  Channel
geos                      3.13.0               hac325c4_0    conda-forge

If this continues to cause issue, can you try recreate your conda env using the dev.yml file using the following command, then install PMP, and run the notebook?

$ conda env create -n [YOUR ENV NAME] -f conda-env/dev.yml

# (after go to pmp directory)
$ pip install .

[acordonez]

@lee1043
Here's matplotlib:

(pmp_dev_enso2) bash-4.4$ conda list matplotlib
# packages in environment at /home/ordonez4/miniconda3/envs/pmp_dev_enso2:
#
# Name                    Version                   Build  Channel
matplotlib                3.9.2           py310hff52083_2    conda-forge
matplotlib-base           3.9.2           py310h68603db_2    conda-forge
matplotlib-inline         0.1.7              pyhd8ed1ab_0    conda-forge

I did use mamba create -n [YOUR ENV NAME] -f conda-env/dev.yml and pip to make the environment I tested, so it is using all the versions from dev.yml.

I tried adding basemap to the dependencies in dev.yml and creating another environment, but then I get this conflict:

Could not solve for environment specs
Encountered problems while solving:
  - package enso_metrics-1.1.3-py310hff52083_0 requires cdms2, but none of the providers can be installed

The environment can't be solved, aborting the operation

[lee1043]

@acordonez thank you, I was able to reproduce the issue on my end as well.

$  conda install conda-forge::basemap

Channels:
 - conda-forge
 - defaults
Platform: osx-64
Collecting package metadata (repodata.json): done
Solving environment: | warning  libmamba Added empty dependency for problem type SOLVER_RULE_UPDATE
failed

LibMambaUnsatisfiableError: Encountered problems while solving:
  - cannot install both pin-1-1 and pin-1-1
  - nothing provides _numpy_rc needed by numpy-2.1.0rc1-py310he367959_0

Could not solve for environment specs
The following packages are incompatible
├─ basemap is installable with the potential options
│  ├─ basemap 1.2.0 would require
│  │  └─ numpy >=1.9.3,<2.0a0 , which can be installed;
│  ├─ basemap [1.2.0|1.2.1] would require
│  │  └─ numpy >=1.14.6,<2.0a0 , which can be installed;
│  ├─ basemap 1.2.2 would require
│  │  └─ numpy >=1.21.4,<2.0a0 , which can be installed;
│  ├─ basemap [1.2.2|1.3.3|1.3.4|1.3.5|1.3.6] would require
│  │  └─ numpy >=1.21.6,<2.0a0 , which can be installed;
│  ├─ basemap 1.2.2 would require
│  │  └─ numpy >=1.16.5,<2.0a0 , which can be installed;
│  ├─ basemap 1.2.2 would require
│  │  └─ numpy >=1.16.6,<2.0a0 , which can be installed;
│  ├─ basemap 1.2.2 would require
│  │  └─ numpy >=1.19.5,<2.0a0 , which can be installed;
│  ├─ basemap 1.2.2 would require
│  │  └─ numpy >=1.18.5,<2.0a0 , which can be installed;
│  ├─ basemap 1.2.2 would require
│  │  └─ numpy >=1.19.2,<2.0a0 , which can be installed;
│  ├─ basemap 1.2.2 would require
│  │  └─ numpy >=1.19.4,<2.0a0 , which can be installed;
│  ├─ basemap 1.3.3 would require
│  │  └─ numpy >=1.22.4,<2.0a0 , which can be installed;
│  ├─ basemap 1.3.4 would require
│  │  └─ numpy >=1.23.2,<2.0a0 , which can be installed;
│  ├─ basemap 1.3.4 would require
│  │  └─ numpy >=1.23.1,<2.0a0 , which can be installed;
│  ├─ basemap [1.3.5|1.3.6] would require
│  │  └─ numpy >=1.23.4,<2.0a0 , which can be installed;
│  ├─ basemap 1.3.6 would require
│  │  └─ numpy >=1.23.5,<2.0a0 , which can be installed;
│  ├─ basemap 1.3.6 would require
│  │  └─ numpy >=1.24.2,<2.0a0 , which can be installed;
│  ├─ basemap 1.3.7 would require
│  │  └─ numpy >=1.24.3,<2.0a0 , which can be installed;
│  ├─ basemap 1.3.8 would require
│  │  └─ numpy >=1.25.2,<2.0a0 , which can be installed;
│  ├─ basemap [1.3.8|1.3.9|1.4.0|1.4.1] would require
│  │  └─ numpy >=1.24.4,<2.0a0 , which can be installed;
│  ├─ basemap 1.3.9 would require
│  │  └─ numpy >=1.26.2,<2.0a0 , which can be installed;
│  ├─ basemap 1.4.0 would require
│  │  └─ numpy >=1.26.3,<2.0a0 , which can be installed;
│  └─ basemap 1.4.1 would require
│     └─ numpy >=1.26.4,<2.0a0 , which can be installed;
├─ numpy >=2.0.0,<3.0.0  is not installable because there are no viable options
│  ├─ numpy [2.0.0|2.0.1|...|2.1.3] conflicts with any installable versions previously reported;
│  └─ numpy [2.1.0rc1|2.2.0rc1] would require
│     └─ _numpy_rc, which does not exist (perhaps a missing channel);
└─ pin-1 is installable with the potential options
   ├─ pin-1 1, which can be installed;
   └─ pin-1 1 conflicts with any installable versions previously reported.

Pins seem to be involved in the conflict. Currently pinned specs:
 - python 3.10.* (labeled as 'pin-1')

It looks like the issue is mostly from conflict between numpy>2.0 and basemap (that is used by enso_metrics). The issue is known at matplotlib/basemap#604 but it's hard to predict their timeline to get this fixed from basemap.

@yyplanton should we consider move to cartopy instead of basemap?

[lee1043]

Updating ENSO plot to use cartopy instead basemap is in progress: CLIVAR-PRP/ENSO_metrics#88. Once this merged and released, enso metrics version 1.1.4 will be required from PMP.

[lee1043]

For enso metrics, CLIVAR-PRP/ENSO_metrics#88 merged and new version tagged (v1.1.4). I am reassign this PR for PMP v3.8.

[lee1043]

@acordonez I think this PR is now ready for another round of review -- thank you for doing this! Basemap is replaced by cartopy in the new enso_metrics version, v1.1.4. Could you rerun the demo 6 in the update env and let me know if any further issue arise?

[acordonez]

@lee1043 Just want to check, is it expected to get null values in the demo JSON results? For example:

"BiasPrLatRmse": {
    "diagnostic": {
      "ACCESS1-0_r1i1p1": {
        "value": null,
        "value_error": null
      },
      "ERA-Interim": {
        "value": null,
        "value_error": null
      },
      "GPCPv2.3": {
        "value": null,
        "value_error": null
      }
    },

[lee1043]

@acordonez thank you for checking. Could you check if value under metric key got numbers like below?

                    "value": {
                        "BiasPrLatRmse": {
                            "diagnostic": {
                                "ACCESS1-0_r1i1p1": {
                                    "value": null,
                                    "value_error": null
                                },
                                "ERA-Interim": {
                                    "value": null,
                                    "value_error": null
                                },
                                "GPCPv2.3": {
                                    "value": null,
                                    "value_error": null
                                }
                            },
                            "metric": {
                                "ERA-Interim": {
                                    "value": 1.107530462384606,
                                    "value_error": null
                                },
                                "GPCPv2.3": {
                                    "value": 1.9301242972172137,
                                    "value_error": null
                                }
                            }
                        },

[acordonez]

@lee1043 Yes the two metrics values match. Otherwise this ran without issue for me.

[lee1043]

@acordonez nice, thank you!