Merge pull request #369 from timtroendle/fix-arm-compatibility

Fix Linux compatibility without breaking ARM macOS compatibility

.github/workflows/environments.yaml

@@ -0,0 +1,45 @@
+name: Tests of conda environments on both Linux and Mac
+
+on:
+    push:
+        branches:
+            - main
+            - develop
+        paths:
+           - envs/*.yaml
+           - templates/environment.yaml
+           - requirements-*.yaml
+    pull_request:
+        paths:
+           - envs/*.yaml
+           - templates/environment.yaml
+           - requirements-*.yaml
+jobs:
+  environments-are-installable:
+    name: Environments are installable
+    runs-on: ${{ matrix.os }}
+    strategy:
+        fail-fast: false
+        matrix:
+            os: [ubuntu-latest, macos-latest]
+            env:
+                - ./envs/geo.yaml
+                - ./envs/default.yaml
+                - ./envs/geo.yaml
+                - ./envs/hydro.yaml
+                - ./envs/shell.yaml
+                - ./envs/test.yaml
+                - ./envs/vis.yaml
+                - ./requirements-docs.yaml
+                - ./requirements-test.yaml
+                - ./templates/environment.yaml
+    defaults:
+        run:
+            shell: bash -l {0}
+    steps:
+      - uses: actions/checkout@v4
+      - uses: conda-incubator/setup-miniconda@v3
+        with:
+            miniconda-version: latest
+      - name:  ${{ matrix.env }} env
+        run: conda env create -f ${{ matrix.env }} -n environment_test --dry-run

CHANGELOG.md

@@ -27,6 +27,7 @@
 * **ADD** ability to run on Apple silicon devices (#263).
     * Updated geo packages from gdal 3.2 -> 3.3.
 * **ADD** re-execution triggers based on config and env changes (#264).
+* **ADD** continuous integration test of all conda environments on both ARM macOS and Linux (#369).
 
 ### Updated (models)
 
@@ -40,21 +41,31 @@
 
 ### Updated (workflow)
 
+* **UPDATE** environments to fix issues on Linux and Macos-arm64 (#357, #369):
+    * libnetcdf=4.8.1
+    * netCDF4=1.6.2
+    * hdf5=1.12.2
+* **UPDATE** geo, hydro, and test-eurocalliope environments to handle libnetcdf=4.8.1 (#369):
+    * gdal=3.6.2
+    * libgdal=3.6.2
+    * fiona=1.9.1
+    * rasterio=1.3.6
+    * geopandas=0.13.2
+    * shapely=1.8.5
 * **UPDATED** to new Zenodo source of land-use potentials, to include potentials at the [e-Highway 2050 project](https://cordis.europa.eu/project/id/308908/reporting) resolution (`ehighways`) (#370).
 * **UPDATED** structure of YAML templates and parametrisation:
     * Parametrisation moved to eurocalliopelib.
     * Rules to parametrise split into smaller technology-specific rules, to ensure inputs are directly relevant to the files being parametrised.
     * YAML templates restructured to match structure of final model (see `Updated (models) above`);
-
 * **UPDATE** cluster sync infrastructure to retain file permission defaults on the cluster. This change improves team collaboration, as default group settings will apply to the files on the cluster (#214).
 * **UPDATE** the declaration of required cluster resources. Moving away from a mechanism that is deprecated in Snakemake (#211).
 * **UPDATE** default Snakemake profile to be activated automatically, for convenience (#264, #268).
 * **UPDATE** default conda prefix directory including consistent handling of the path to eurocalliopelib (#264, #331).
 * **UPDATE** Snakemake to v8.10.7 (#330)
     * Ensures that conda environment builds ignore default package specifications (#289).
     * Fixes localrules through integration of new `localrule` directive (#368).
-
 * **UPDATE** source of fraction of shared coast for offshore wind capacity factor distribution from a fixed shape download to an internal rule which can handle ad hoc shapes (partial #238).
+* **UPDATE** dropped support for Intel macOS. The workflow may still run on Intel macOS, but we do not actively maintain support (#369).
 * **UPDATE** link to GADM data following changes upstream (#376).
 
 ### Fixed (models)
@@ -70,7 +81,6 @@
 * **FIX** fixed optimisation tolerance of hydro power plants from xtol to xatol (#266).
 * **FIX** source of Exclusive Economic Zones (EEZ) to use a cache on [zenodo](https://sandbox.zenodo.org/records/45135) so that we can keep using v11 (#332).  FIXME: update to actual zenodo record before next Euro-Calliope release.
 * **FIX** fixed rule `download_basins_database`, which previously failed on some linux and mac machines, by requiring a more recent curl in the environment `envs/shell.yaml` (#267).
-* **FIX** pin `h5py`, `hdf5` and `libnetcdf` in all environments which rely on `xarray`, to prevent issues on linux-x86 (#357).
 
 ## 1.1.0 (2021-12-22)
 

docs/workflow/build.md

@@ -10,7 +10,7 @@ The build process will require an internet connection, several gigabytes of avai
 ## Prepare
 
 !!! Info
-    The workflow is developed on macOS, tested on macOS and Linux, but it cannot run natively on Windows.
+    The workflow is developed mostly on ARM macOS, tested on ARM macOS and Linux, but it cannot run natively on Windows.
 
 1. Download your copy of the latest Euro-Calliope release: [![workflow DOI](https://img.shields.io/badge/workflow-10.5281/zenodo.3949793-blue)](https://doi.org/10.5281/zenodo.3949793) (or [clone any version from GitHub](https://github.com/calliope-project/euro-calliope)).
 

envs/default.yaml

@@ -5,7 +5,7 @@ dependencies:
     - python=3.9
     - ipython=7.21.0
     - ipdb=0.13.13
-    - numpy=1.20.2
+    - numpy=1.20.3
     - pandas=1.2.3
     - bottleneck=1.3.2
     - xlrd=2.0.1
@@ -14,10 +14,9 @@ dependencies:
     - jinja2=2.11.3
     - pip=21.0.1
     - xarray=0.17.0
-    - netCDF4=1.5.6
-    - hdf5=1.10
-    - h5py=3.1.0
-    - libnetcdf=4.8.0
+    - netCDF4=1.6.2
+    - hdf5=1.12.2
+    - libnetcdf=4.8.1
     - pip:
         - -e ./lib
         - styleframe==4.2

envs/geo.yaml

@@ -4,24 +4,24 @@ channels:
 dependencies:
     - python=3.9
     - ipdb=0.13.13
-    - numpy=1.20.2
+    - numpy=1.20.3
     - scipy=1.6.2
     - pandas=1.2.3
-    - gdal=3.3.1
-    - libgdal=3.3.1
-    - fiona=1.8.20
-    - rasterio=1.2.6
+    - gdal=3.6.2
+    - libgdal=3.6.2
+    - fiona=1.9.1
+    - rasterio=1.3.6
     - rasterstats=0.14.0
-    - geos=3.9.1
-    - geopandas=0.9.0
-    - netcdf4=1.5.6
+    - geos=3.11.1
+    - geopandas=0.13.2
     - xarray=0.17.0
+    - netCDF4=1.6.2
+    - hdf5=1.12.2
+    - libnetcdf=4.8.1
     - jinja2=2.11.3
     - networkx=2.5
     - pycountry=18.12.8
     - pip=21.0.1
-    - hdf5=1.10
-    - h5py=3.1.0
-    - libnetcdf=4.8.0
+    - shapely=1.8.5
     - pip:
         - -e ./lib[geo]

envs/hydro.yaml

@@ -4,23 +4,24 @@ channels:
 dependencies:
     - python=3.9
     - ipdb=0.13.13
-    - numpy=1.20.2
+    - numpy=1.20.3
     - pandas=1.2.3
-    - gdal=3.3.1
-    - fiona=1.8.20
-    - rasterio=1.2.6
-    - geopandas=0.9.0
-    - netcdf4=1.5.6
+    - gdal=3.6.2
+    - libgdal=3.6.2
+    - fiona=1.9.1
+    - rasterio=1.3.6
+    - geopandas=0.13.2
     - xarray=0.16.2
+    - netCDF4=1.6.2
+    - hdf5=1.12.2
+    - libnetcdf=4.8.1
     - dask=2021.3.1
     - bottleneck=1.3.2
     - cdsapi=0.5.1
     - pycountry=18.12.8
     - atlite=0.2.1
-    - pip=21.0.1
-    - hdf5=1.10
-    - h5py=3.1.0
-    - libnetcdf=4.7
     - scipy=1.9.1  # sub-dependency of atlite. Update when updating numpy.
+    - pip=21.0.1
+    - shapely=1.8.5
     - pip:
         - -e ./lib[geo]

envs/test.yaml

@@ -13,6 +13,6 @@ dependencies:
     - pytest-html=3.2.0
     - calliope=0.6.10
     - pyomo=6.4.1
-    - hdf5=1.10
-    - h5py=3.1.0
-    - libnetcdf=4.8.0
+    - netCDF4=1.6.2
+    - hdf5=1.12.2
+    - libnetcdf=4.8.1

lib/eurocalliopelib/geo/spatiotemporal.py

@@ -1,6 +1,7 @@
 import geopandas as gpd
 import numpy as np
 import pandas as pd
+import pyproj
 import xarray as xr
 from geopandas.tools import overlay
 from shapely.geometry import Point
@@ -42,7 +43,7 @@ def area_weighted_time_series(shapes, spatiotemporal, gridcell_overlap_threshold
 def assert_correct_form(shapes, spatiotemporal):
     assert shapes.crs
     assert "crs" in spatiotemporal.attrs
-    assert shapes.crs == gpd.tools.crs.CRS(spatiotemporal.attrs["crs"])
+    assert shapes.crs == pyproj.CRS(spatiotemporal.attrs["crs"])
     assert "y" in spatiotemporal.dims, "Expect dimension 'y'"
     assert "x" in spatiotemporal.dims, "Expect dimension 'x'"
     assert "timestep" in spatiotemporal.dims, "Expect dimension 'timestep'"

requirements-test.yaml

@@ -3,22 +3,25 @@ channels:
     - conda-forge
 dependencies:
     - python=3.8
-    - numpy=1.20.2
+    - numpy=1.20.3
     - scipy=1.6.2
     - pandas=1.2.3
-    - gdal=3.3.1
-    - libgdal=3.3.1
-    - fiona=1.8.20
-    - rasterio=1.2.6
+    - gdal=3.6.2
+    - libgdal=3.6.2
+    - fiona=1.9.1
+    - rasterio=1.3.6
     - rasterstats=0.14.0
-    - geos=3.9.1
-    - geopandas=0.9.0
+    - geos=3.11.1
+    - geopandas=0.13.2
     - jinja2=2.11.3
-    - netcdf4=1.5.6
     - xarray=0.17.0
+    - netCDF4=1.6.2
+    - hdf5=1.12.2
+    - libnetcdf=4.8.1
     - pycountry=18.12.8
     - pytest=7.3.1
     - pytest-html=3.2.0
     - pip=21.0.1
+    - shapely=1.8.5
     - pip:
         - -e ./lib[geo]

scripts/shapes/gadm.py

@@ -74,7 +74,7 @@ def _in_study_area(feature):
 
 
 def _all_parts_in_study_area(feature, study_area):
-    unit = _to_multi_polygon(feature["geometry"])
+    unit = _to_multi_polygon(shapely.geometry.shape(feature["geometry"]))
     if not study_area.contains(unit):
         print(f"Removing parts of {_feature_name(feature)} outside of study area.")
         new_unit = shapely.geometry.MultiPolygon([

scripts/shapes/nuts.py

@@ -98,7 +98,8 @@ def _fix_country_feature(feature):
 
 
 def _all_parts_in_study_area_and_crs(feature, src_crs, dst_crs, study_area):
-    unit = _to_multi_polygon(feature["geometry"])
+    unit = _to_multi_polygon(shapely.geometry.shape(feature["geometry"]))
+
     if not study_area.contains(unit):
         print(
             "Removing parts of {} outside of study area.".format(

templates/environment.yaml

@@ -7,6 +7,9 @@ dependencies:
     - numpy=1.23
     - pandas=1.5
     - xarray=2022.3
+    - netCDF4=1.6.2
+    - hdf5=1.12.2
+    - libnetcdf=4.8.1
     - gurobi=9.5.1
     - calliope=0.6.10
     - pyomo=6.4.1