Merge pull request #264 from timtroendle/feature-update-workflow-infra

Update workflow infrastructure

.github/workflows/pythonpackage.yaml

@@ -22,7 +22,7 @@ jobs:
                 auto-update-conda: true
                 python-version: 3.8
                 activate-environment: test-eurocalliope
-                environment-file: test-requirements.yaml
+                environment-file: requirements-test.yaml
             - name: Test with pytest
               run: |
                 pytest

.github/workflows/schemavalidation.yaml

@@ -14,7 +14,7 @@ jobs:
         steps:
             - uses: actions/checkout@v3
             - name: Set up Python
-              uses: actions/setup-python@v2
+              uses: actions/setup-python@v4
               with:
                   python-version: 3.8
             - name: Install dependencies

.readthedocs.yaml

@@ -3,9 +3,11 @@
 # See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
 
 version: 2
-
+build:
+  os: ubuntu-22.04 # required
+  tools:
+    python: mambaforge-4.10
 mkdocs:
   configuration: mkdocs.yaml
-
 conda:
-  environment: envs/docs.yaml
+  environment: requirements-docs.yaml

CHANGELOG.md

@@ -16,6 +16,7 @@
 * **ADD** a Snakemake rule that generates a .csv and .nc file that provide an summary of the potentials (= per-tech constraints) for each technology and location (#250).
 * **ADD** ability to run on Apple silicon devices (#263).
     * Updated geo packages from gdal 3.2 -> 3.3.
+* **ADD** re-execution triggers based on config and env changes (#264).
 
 ### Updated (models)
 
@@ -36,6 +37,8 @@
 
 * **UPDATE** cluster sync infrastructure to retain file permission defaults on the cluster. This change improves team collaboration, as default group settings will apply to the files on the cluster (#214).
 * **UPDATE** the declaration of required cluster resources. Moving away from a mechanism that is deprecated in Snakemake (#211).
+* **UPDATE** default Snakemake profile to be activated automatically, for convenience (#264).
+* **UPDATE** default conda prefix directory including consistent handling of the path to eurocalliopelib (#264).
 
 ### Fixed (models)
 
@@ -45,10 +48,6 @@
 
 * **FIX** links in the documention to always point to the most recent version of the pre-builts (#218).
 
-### Fixed (environments)
-
-* **FIX** incompatibility of Snakemake 6.1.1 and tabulate 0.9 by downgrading to tabulate 0.8.10. (#249).
-
 ## 1.1.0 (2021-12-22)
 
 This version is a minor update to v1.0. It comes with updated input data, updated dependencies, several convenience features, several optional overrides, and with massively extended documentation. You should not expect model results to deviate much from v1.0.

CONTRIBUTING.md

@@ -47,8 +47,8 @@ The following subsections show you how to run the tests and how to write your ow
 Tests of models with continental and national resolution run automatically when you run the entire workflow. To run the tests of models with regional resolution too, do the following:
 
 ```bash
-snakemake --profile profiles/default # builds all models and runs continental and national tests
-snakemake --profile profiles/default all_tests # builds all models and runs continental, national, and regional tests
+snakemake # builds all models and runs continental and national tests
+snakemake all_tests # builds all models and runs continental, national, and regional tests
 ```
 
 ### Run the minimal workflow tests
@@ -57,7 +57,7 @@ As a developer, you may want to run the entire workflow often to spot errors ear
 For that, you can use a minimal test configuration that takes less time to run than the default configuration.
 
 ```bash
-snakemake --profile profiles/default --configfile="config/minimal.yaml" all_tests
+snakemake --configfile="config/minimal.yaml" all_tests
 ```
 
 Make sure to run this in a clean working directory.
@@ -71,7 +71,7 @@ Here's how.
 First, create a test environment using mamba or conda:
 
 ```bash
-mamba env create -f test-requirements.yaml --no-default-packages # or replace mamba with conda
+mamba env create -f requirements-test.yaml --no-default-packages # or replace mamba with conda
 conda activate test-eurocalliope
 ```
 
@@ -121,7 +121,7 @@ Be aware that you can publish a release only if you are a maintainer of the GitH
     3. Update the changelog and add the release date.
     4. Update the release date in [./CITATION.cff](./CITATION.cff).
     5. (If necessary) Update [./LICENSE.md](./LICENSE.md).
-    6. (If necessary) Update `docs/img/spatial-scope-and-resolutions.png` by running `snakemake --profile profiles/default -s rules/doc.smk docs/img/spatial-scope-and-resolutions.png`. Inspect the result visually. Check it in if it changed; check out the old version if it did not change. The figure will change when the spatial scope or resolution has changed.
+    6. (If necessary) Update `docs/img/spatial-scope-and-resolutions.png` by running `snakemake -s rules/doc.smk docs/img/spatial-scope-and-resolutions.png`. Inspect the result visually. Check it in if it changed; check out the old version if it did not change. The figure will change when the spatial scope or resolution has changed.
 2. Build the pre-builts and test everything using the `all_tests` rule. Make sure you start with a clean workflow folder: delete `./build` *and* `./data/automatic` should they exist.
 3. Commit, open a pull request onto `develop`, and merge the release branch into both `develop` and `main` after successful review.
 4. Add a `vX.Y.Z` release tag to `main`, push it, and add a release on GitHub.

README.md

@@ -29,7 +29,7 @@ If you are unable to access the full documentation via ReadTheDocs following the
 
 
 ```bash
-conda env create -f envs/docs.yaml
+conda env create -f requirements-docs.yaml
 conda activate docs
 mkdocs build --no-directory-urls
 ```

Snakefile

@@ -1,14 +1,13 @@
 import glob
 from pathlib import Path
 
-from snakemake.utils import validate
+from snakemake.utils import validate, min_version, makedirs
 
 configfile: "config/default.yaml"
 validate(config, "config/schema.yaml")
 
 root_dir = config["root-directory"] + "/" if config["root-directory"] not in ["", "."] else ""
 __version__ = open(f"{root_dir}VERSION").readlines()[0].strip()
-script_dir = f"{root_dir}scripts/"
 test_dir = f"{root_dir}tests/"
 model_test_dir = f"{test_dir}model"
 template_dir = f"{root_dir}templates/"
@@ -23,6 +22,7 @@ include: "./rules/transmission.smk"
 include: "./rules/demand.smk"
 include: "./rules/nuclear.smk"
 include: "./rules/sync.smk"
+min_version("7.8")
 localrules: all, clean
 wildcard_constraints:
         resolution = "continental|national|regional"
@@ -37,6 +37,17 @@ ALL_CF_TECHNOLOGIES = [
 ]
 ALL_DEMAND_CARRIERS = ["electricity"]
 
+def ensure_lib_folder_is_linked():
+    if not workflow.conda_prefix:
+        return
+    link = Path(workflow.conda_prefix) / "lib"
+    if not link.exists():
+        print("Creating link from conda env dir to eurocalliopelib.")
+        makedirs(workflow.conda_prefix)
+        shell(f"ln -s {workflow.basedir}/lib {workflow.conda_prefix}/lib")
+
+ensure_lib_folder_is_linked()
+
 onstart:
     shell("mkdir -p build/logs")
 onsuccess:
@@ -93,7 +104,6 @@ rule dummy_tech_locations_template:  # needed to provide `techs_and_locations_te
 rule techs_and_locations_template:
     message: "Create {wildcards.resolution} definition file for the {wildcards.tech_group} tech `{wildcards.tech}`."
     input:
-        script = script_dir + "template_techs.py",
         template = techs_template_dir + "{tech_group}/{tech}.yaml",
         locations = "build/data/{resolution}/{tech_group}/{tech}.csv"
     params:
@@ -132,7 +142,6 @@ rule no_params_template:
 rule model_template:
     message: "Generate top-level {wildcards.resolution} model configuration file from template"
     input:
-        script = script_dir + "template_model.py",
         template = model_template_dir + "example-model.yaml",
         non_model_files = expand(
             "build/models/{template}", template=["environment.yaml", "README.md"]
@@ -179,7 +188,6 @@ rule model_template:
 rule build_metadata:
     message: "Generate build metadata."
     input:
-        script_dir + "metadata.py",
         "build/models/continental/example-model.yaml",
         "build/models/national/example-model.yaml",
         "build/models/regional/example-model.yaml",
@@ -218,6 +226,7 @@ rule test:
         runtime = 240
     script: "./tests/model/test_runner.py"
 
+
 rule summarise_potentials:
     message: "Generates netcdf and csv file with potentials for each technology."
     input:

docs/workflow/build-remote.md

@@ -20,15 +20,15 @@ We support this workflow through three Snakemake rules: `send`, `receive`, and `
 It works like the following.
 
 First, start local and make sure the `cluster-sync` configuration parameters fit your environment.
-Next, run `snakemake --profile profiles/default send` to send the entire repository to your cluster.
+Next, run `snakemake send` to send the entire repository to your cluster.
 On the cluster, execute the workflow with Snakemake ([see above](./build-remote.md#build-on-a-cluster)).
-After the workflow has finished, download results by locally running `snakemake --profile profiles/default receive`.
+After the workflow has finished, download results by locally running `snakemake receive`.
 By default, this will download results into `build/cluster`.
 
 This workflow works iteratively too.
-After analysing your cluster results locally, you may want to make changes locally, send these changes to the cluster (`snakemake --profile profiles/default send`), rerun on the cluster, and download updated results (`snakemake --profile profiles/default receive`).
+After analysing your cluster results locally, you may want to make changes locally, send these changes to the cluster (`snakemake send`), rerun on the cluster, and download updated results (`snakemake receive`).
 
-To remove cluster results on your local machine, run `snakemake --profile profiles/default clean_cluster_results`.
+To remove cluster results on your local machine, run `snakemake clean_cluster_results`.
 
 ## Be notified of build successes or fails
 

docs/workflow/build.md

@@ -22,7 +22,7 @@ Using either one, you can create the environment:
         # using mamba
         mamba env create -f environment.yaml --no-default-packages
         conda activate euro-calliope
-        snakemake --profile profiles/default --list # test your installation
+        snakemake --list # test your installation
 
         # using conda
         conda env create -f environment.yaml --no-default-packages
@@ -39,7 +39,7 @@ By default, the entire model will be built in the `./build/model` folder when yo
 Execute the workflow like so:
 
 ```bash
-snakemake --profile profiles/default
+snakemake
 ```
 
 ## Run

environment.yaml

@@ -3,7 +3,8 @@ channels:
     - conda-forge
     - bioconda
 dependencies:
-    - python=3.8
+    - python=3.11
     - pycountry=18.12.8
-    - snakemake-minimal=6.1.1
-    - tabulate=0.8.10 # fixes incompatibility of tabulate 0.9 and snakemake 6.1.1. Should be fixed after snakemake 7.15.2
+    - snakemake-minimal=7.26.0
+variables:
+    SNAKEMAKE_PROFILE: ./profiles/default

envs/default.yaml

@@ -12,4 +12,4 @@ dependencies:
     - jinja2=2.11.3
     - pip=21.0.1
     - pip:
-        - -e ../../lib
+        - -e ./lib

envs/geo.yaml

@@ -20,4 +20,4 @@ dependencies:
     - pycountry=18.12.8
     - pip=21.0.1
     - pip:
-        - -e ../../lib[geo]
+        - -e ./lib[geo]

envs/hydro.yaml

@@ -18,4 +18,4 @@ dependencies:
     - atlite=0.2.1
     - pip=21.0.1
     - pip:
-        - -e ../../lib[geo]
+        - -e ./lib[geo]

envs/vis.yaml

@@ -17,4 +17,4 @@ dependencies:
     - pycountry=18.12.8
     - pip=21.0.1
     - pip:
-        - -e ../../lib[geo]
+        - -e ./lib[geo]

lib/README.md

@@ -12,7 +12,7 @@ The following assumes you are in the root folder of this repository, i.e. the pa
 
 Best install `eurocalliopelib` from the conda environment:
 
-    $ conda env create -f test-requirements.yaml
+    $ conda env create -f requirements-test.yaml
     $ conda activate test-eurocalliope
 
 ### Run the test suite

profiles/conda/config.yaml

@@ -1,3 +1,4 @@
 use-conda: True
 conda-frontend: conda
-cores: 1
+cores: 2
+conda-prefix: ../envs/snakemake/euro-calliope/

profiles/default/config.yaml

@@ -1,2 +1,4 @@
 use-conda: True
-cores: 1
+conda-frontend: mamba
+cores: 2
+conda-prefix: ../envs/snakemake/euro-calliope/

profiles/euler/config.yaml

@@ -4,6 +4,7 @@ local-cores: 1
 latency-wait: 60
 use-envmodules: True
 use-conda: True
+conda-prefix: ../envs/snakemake/euro-calliope/
 default-resources: [
     "runtime=10",
     "cores=1",

envs/docs.yaml → requirements-docs.yaml

@@ -9,8 +9,8 @@ dependencies:
     - graphviz=2.48.0
     - pydot=1.4.2
     - pycountry=18.12.8
-    - snakemake-minimal=6.1.1
+    - snakemake-minimal=7.26.0
     - pip=21.0.1
     - pip:
         - jsonschema2md==0.2.1
-        - -e ../lib[docs]
+        - -e ./lib[docs]

rules/biofuels.smk

@@ -14,7 +14,6 @@ rule download_biofuel_potentials_and_costs:
 rule preprocess_biofuel_potentials_and_cost:
     message: "Extract national potentials and cost from raw biofuel data."
     input:
-        script = script_dir + "biofuels/extract.py",
         potentials_and_costs = rules.download_biofuel_potentials_and_costs.output[0]
     params:
         feedstocks = {
@@ -32,7 +31,6 @@ rule preprocess_biofuel_potentials_and_cost:
 rule biofuels:
     message: "Determine biofuels potential on {wildcards.resolution} resolution for scenario {wildcards.scenario}."
     input:
-        script = script_dir + "biofuels/allocate.py",
         units = rules.units_without_shape.output[0],
         land_cover = rules.potentials.output.land_cover,
         population = rules.potentials.output.population,
@@ -58,7 +56,6 @@ rule biofuels:
 rule bio_techs_and_locations_template:
     message: "Create biofuel tech definition file from template."
     input:
-        script = script_dir + "biofuels/template_bio.py",
         template = techs_template_dir + "supply/biofuel.yaml",
         biofuel_cost = "build/data/regional/biofuel/{scenario}/costs-eur-per-mwh.csv".format(
             scenario=config["parameters"]["jrc-biofuel"]["scenario"]

rules/demand.smk

@@ -14,7 +14,6 @@ rule download_raw_load:
 rule electricity_load_national:
     message: "Preprocess raw electricity load data and retrieve load time series per country."
     input:
-        script = script_dir + "demand/national_load.py",
         load = rules.download_raw_load.output[0]
     params:
         first_year = config["scope"]["temporal"]["first-year"],
@@ -29,7 +28,6 @@ rule electricity_load_national:
 rule electricity_load:
     message: "Generate electricity load time series for every location on {wildcards.resolution} resolution."
     input:
-        script = script_dir + "demand/load.py",
         units = rules.units.output[0],
         demand_per_unit = rules.potentials.output.demand,
         national_load = rules.electricity_load_national.output[0]

rules/doc.smk

@@ -7,7 +7,6 @@ configfile: "./config/default.yaml"
 rule spatial_scope_and_resolutions:
     message: "Plot spatial scope and resolution for documentation."
     input:
-        script = "scripts/vis/spatial_scope_and_resolutions.py",
         regional_units = "build/data/regional/units.geojson",
         national_units = "build/data/national/units.geojson"
     params: