Merge pull request #361 from timtroendle/feature-workflow-profile

Update default profile to be a workflow-profile

CHANGELOG.md

@@ -44,9 +44,9 @@
 
 * **UPDATE** cluster sync infrastructure to retain file permission defaults on the cluster. This change improves team collaboration, as default group settings will apply to the files on the cluster (#214).
 * **UPDATE** the declaration of required cluster resources. Moving away from a mechanism that is deprecated in Snakemake (#211).
-* **UPDATE** default Snakemake profile to be activated automatically, for convenience (#264).
+* **UPDATE** default Snakemake profile to be activated automatically, for convenience (#264, #268).
 * **UPDATE** default conda prefix directory including consistent handling of the path to eurocalliopelib (#264, #331).
-* **UPDATE** snakemake to v8.10.6 (#330), which ensures that conda environment builds ignore default package specifications (#289).
+* **UPDATE** snakemake to v8.10.7 (#330), which ensures that conda environment builds ignore default package specifications (#289).
 
 ### Fixed (models)
 

CONTRIBUTING.md

@@ -118,13 +118,19 @@ Be aware that you can publish a release only if you are a maintainer of the GitH
         * [./lib/eurocalliopelib/__init__.py](./lib/eurocalliopelib/__init__.py)
         * [./templates/README.md](./templates/README.md)
         * [./docs/index.md](./docs/index.md)
-    2. Verify consistent versions of Snakemake and Calliope.
-    Their versions are spread over the entire repository (including urls in the documentation).
-    Use search and replace to make sure all versions are consistent.
-    3. Update the changelog and add the release date.
-    4. Update the release date in [./CITATION.cff](./CITATION.cff).
-    5. (If necessary) Update [./LICENSE.md](./LICENSE.md).
-    6. (If necessary) Update `docs/img/spatial-scope-and-resolutions.png` by running `snakemake -s rules/doc.smk docs/img/spatial-scope-and-resolutions.png`. Inspect the result visually. Check it in if it changed; check out the old version if it did not change. The figure will change when the spatial scope or resolution has changed.
+    2. Verify consistent versions of Snakemake.
+        * [./environment.yaml](./environment.yaml)
+        * [./requirements-docs.yaml](./requirements-docs.yaml)
+        * [./Snakefile](./Snakefile)
+    3. Verify consistent versions of Calliope.
+        * [./envs/test.yaml](./envs/test.yaml)
+        * [./templates/environment.yaml](./templates/environment.yaml)
+        * [./templates/models/example-model.yaml](./templates/models/example-model.yaml)
+        * [./mkdocs.yaml](./mkdocs.yaml)
+    4. Update the changelog and add the release date.
+    5. Update the release date in [./CITATION.cff](./CITATION.cff).
+    6. (If necessary) Update [./LICENSE.md](./LICENSE.md).
+    7. (If necessary) Update `docs/img/spatial-scope-and-resolutions.png` by running `snakemake -s rules/doc.smk docs/img/spatial-scope-and-resolutions.png`. Inspect the result visually. Check it in if it changed; check out the old version if it did not change. The figure will change when the spatial scope or resolution has changed.
 2. Build the pre-builts and test everything using the `all_tests` rule. Make sure you start with a clean workflow folder: delete `./build` *and* `./data/automatic` should they exist.
 3. Commit, open a pull request onto `develop`, and merge the release branch into both `develop` and `main` after successful review.
 4. Add a `vX.Y.Z` release tag to `main`, push it, and add a release on GitHub.

Snakefile

@@ -26,8 +26,7 @@ include: "./rules/nuclear.smk"
 include: "./rules/transport.smk"
 include: "./rules/sync.smk"
 include: "./rules/heat.smk"
-
-min_version("7.8")
+min_version("8.10")
 localrules: all, clean
 wildcard_constraints:
         resolution = "continental|national|regional"

docs/model/customisation.md

@@ -15,13 +15,13 @@ You have the following three options:
 With the Calliope model in your hands, you will be able to change any model parameter, any technology specifics, and the model definition to your liking.
 This kind of customisation can be useful to get to know the model and its parameters.
 To create reliable results, we advise making manual changes only to the model definition (`example-model.yaml`) as this makes it possible to trace those changes later.
-A typical customisation here would be to change the solver from `gurobi` to an open-source solver, e.g. `cbc` (see [Calliope's documentation](https://calliope.readthedocs.io/en/v0.6.10/user/config_defaults.html#run-configuration)).
+A typical customisation here would be to change the solver from `gurobi` to an open-source solver, e.g. `cbc` (see [Calliope's documentation](https://calliope.readthedocs.io/en/v{{ calliope_version }}/user/config_defaults.html#run-configuration)).
 We consider all Euro-Calliope model subcomponents (everything other than the model definition itself) as a toolbox from which you can choose to define your model -- see the [Import customisation option](./customisation.md#importing-modules).
 
 ## Importing modules
 
 The `example-model.yaml` definition file in each resolution sub-directory (e.g. `national/example-model.yaml`) specifies a list of other files to bring together to describe the model (under the `import` key).
-This list can be changed by the modeller to select a combination of different files (see also [Calliope's documentation](https://calliope.readthedocs.io/en/v0.6.10/user/building.html#files-that-define-a-model)).
+This list can be changed by the modeller to select a combination of different files (see also [Calliope's documentation](https://calliope.readthedocs.io/en/v{{ calliope_version }}/user/building.html#files-that-define-a-model)).
 These files represent "modules" of the model definition and contain everything necessary for a given technology or technology group to exist.
 For instance, `techs/supply/hydro.yaml` defines two technologies (under the `techs` key) which will convert river flows into electricity.
 It also places that technology in every relevant modelled location (under the `locations` key), along with any location-specific information that is needed; in this case, the maximum capacity of hydropower in that location.
@@ -130,7 +130,7 @@ Here, we describe each module in terms of the technologies they contain (`callio
         This supply has high variable cost (see `tech-cost.yaml` parameter file) and no fixed cost.
         Due to its high cost, it will only be used when no other, less costly, option is available.
 
-        Calliope provides a built-in mechanism that is similar: [`ensure-feasibility`](https://calliope.readthedocs.io/en/v0.6.10/user/building.html#allowing-for-unmet-demand).
+        Calliope provides a built-in mechanism that is similar: [`ensure-feasibility`](https://calliope.readthedocs.io/en/v{{ calliope_version }}/user/building.html#allowing-for-unmet-demand).
         The benefit of using the `load-shedding` override over Calliope's built-in mechanism is that it is more targeted towards modelling shedding of electrical load and provides more flexibility -- for example in terms of the cost of shed load.
 
 
@@ -204,7 +204,7 @@ Here, we describe each module in terms of the technologies they contain (`callio
 
 ## Overrides and scenarios
 
-Calliope [overrides](https://calliope.readthedocs.io/en/v0.6.10/user/building.html#scenarios-and-overrides) enable models to be easily manipulated.
+Calliope [overrides](https://calliope.readthedocs.io/en/v{{ calliope_version }}/user/building.html#scenarios-and-overrides) enable models to be easily manipulated.
 An override named `freeze-hydro-supply-capacities` can be used for example in this way:
 
 ``` bash
@@ -223,7 +223,7 @@ For instance, `freeze-hydro-supply-capacities` and `freeze-hydro-storage-capacit
 You can also define your own overrides to manipulate any model component.
 We recommend you add these overrides into the model definition YAML file, to ensure they are easy to trace.
 
-In Calliope, [scenarios](https://calliope.readthedocs.io/en/v0.6.7/user/building.html#scenarios-and-overrides) are groups of overrides and/or other scenarios.
+In Calliope, [scenarios](https://calliope.readthedocs.io/en/v{{ calliope_version }}/user/building.html#scenarios-and-overrides) are groups of overrides and/or other scenarios.
 In Euro-Calliope, it can be helpful to define scenarios to help group similar overrides together.
 For instance, cost overrides from the Danish Energy Agency are defined in various files, since they are loaded in alongside the technologies they affect (the option to override offshore wind costs only exists when you load the `techs/supply/wind-offshore.yaml` module).
 You can pre-define scenarios in your model definition file, such as:

docs/model/pre-built.md

@@ -3,7 +3,7 @@
 If you want an easy entry into using and exploring Euro-Calliope, our pre-built models provide the jump start you need.
 The pre-built models are ordinary Calliope models that you can use like any other.
 If you are unfamiliar with Calliope, we'll show you below how to run the model.
-After going through these first steps, we advise you to head over to [Calliope's documentation](https://calliope.readthedocs.io/en/v0.6.10/) to learn its basic usage.
+After going through these first steps, we advise you to head over to [Calliope's documentation](https://calliope.readthedocs.io/en/v{{ calliope_version }}/) to learn its basic usage.
 
 ## Prepare
 

docs/workflow/build-remote.md

@@ -1,7 +1,7 @@
 # Build on a cluster
 
 You may want to build the model on a cluster.
-While you can build Euro-Calliope on [any cluster that is supported by Snakemake](https://snakemake.readthedocs.io/en/v6.1.1/executing/cluster.html), our default configuration is targeted at, and tested on, ETH's Euler cluster.
+While you can build Euro-Calliope on [any cluster that is supported by Snakemake](https://snakemake.readthedocs.io/en/v{{ snakemake_version }}/tutorial/additional_features.html#cluster-or-cloud-execution), our default configuration is targeted at, and tested on, ETH's Euler cluster.
 
 ## Build
 
@@ -11,7 +11,7 @@ To build the model on Euler, use the following command:
 snakemake --profile profiles/euler
 ```
 
-If you want to run on another cluster, read [snakemake's documentation on cluster execution](https://snakemake.readthedocs.io/en/stable/executing/cluster.html) and take `profiles/euler` as a starting point.
+If you want to run on another cluster, read [snakemake's documentation on cluster execution](https://snakemake.readthedocs.io/en/v{{ snakemake_version }}/tutorial/additional_features.html#cluster-or-cloud-execution) and take `profiles/euler` as a starting point.
 
 ## Work local, build on remote
 

docs/workflow/build.md

@@ -42,6 +42,8 @@ Execute the workflow like so:
 snakemake
 ```
 
+This will use Snakemake run parameters from the default `workflow-profile` in `profiles/default`. You may change these parameters and/or use a user defined profile using `--profile=<user-profile>`; see [Snakemake's documentation on profiles](https://snakemake.readthedocs.io/en/stable/executing/cli.html#profiles).
+
 ## Run
 
 The build step creates all individual components of Euro-Calliope, like technologies and time series.
@@ -53,7 +55,7 @@ It is a complete Calliope model and can be used like any other, for example like
 calliope run ./build/models/national/example-model.yaml
 ```
 
-For more information on how to use Calliope models, see [Calliope's documentation](https://calliope.readthedocs.io/en/v0.6.10/).
+For more information on how to use Calliope models, see [Calliope's documentation](https://calliope.readthedocs.io/en/v{{ calliope_version }}/).
 
 ## Customise
 

docs/workflow/customisation.md

@@ -13,19 +13,13 @@ You can, for example, change the temporal and spatial scope of the data and mode
 
 The configuration builds on Snakemake's configuration mechanism and consists of two parts: a default configuration `./config/default.yaml` and a schema declaring all configuration parameters `./config/schema.yaml`.
 To override configuration parameters, you can add another configuration file with just your updates or change parameter values on the command line when calling `snakemake`.
-For details on how the configuration mechanism works, please read [Snakemake's documention](https://snakemake.readthedocs.io/en/v6.1.1/snakefiles/configuration.html).
-
-!!! warning
-    When you change configuration parameters, please consider this important caveat of Snakemake:
-    While Snakemake does understand that parameters have changed, it will not rerun rules with updated parameters by default (see also this [feature request](https://github.com/snakemake/snakemake/issues/976)).
-    To ensure that your configuration parameters are used, you must manually rerun all rules that use the updated parameter.
-    The easiest is to start the build process from scratch by running `snakemake clean`.
+For details on how the configuration mechanism works, please read [Snakemake's documention](https://snakemake.readthedocs.io/en/v{{ snakemake_version }}/snakefiles/configuration.html).
 
 ## Adaptation
 
 Beyond configuration through parameters, you can adapt and extend the workflow in any possible way.
 You can adapt the data pre-processing steps and the way model files are generated, but you can also extend the model by adding your own model files or overrides.
-Customising Euro-Calliope in this way requires a solid understanding of the workflow management system [Snakemake](https://snakemake.readthedocs.io/en/v6.1.1/index.html) that we use.
+Customising Euro-Calliope in this way requires a solid understanding of the workflow management system [Snakemake](https://snakemake.readthedocs.io/en/v{{ snakemake_version }}/index.html) that we use.
 
 Whenever we applied Euro-Calliope in our research we made use of this option.
 Below you will find a list of publications in which we applied Euro-Calliope models.

docs/workflow/overview.md

@@ -2,7 +2,7 @@
 
 Euro-Calliope's workflow builds the models from raw data.
 In essence, it is composed out of three things: scripts that retrieve raw data, process data, and formulate the model; execution environments that explicitly define software dependencies and requirements (equivalent to the conditions in a lab); and the glue ensuring all scripts run in the correct execution environment and the right order.
-Scripts are written in Python; the execution environments are [conda environments](https://docs.conda.io/en/latest/); and the glue holding everything together is the workflow management system [Snakemake](https://snakemake.readthedocs.io/en/v6.1.1/).
+Scripts are written in Python; the execution environments are [conda environments](https://docs.conda.io/en/latest/); and the glue holding everything together is the workflow management system [Snakemake](https://snakemake.readthedocs.io/en/v{{ snakemake_version }}/).
 
 ## Rationale
 

docs/workflow/troubleshooting.md

@@ -7,7 +7,7 @@ running into these problems is completely normal and in many cases a solution is
 Here are some steps we advise you to go through:
 
 1. Read through our frequently asked questions below.
-2. If you did not find an answer to your question, check [Snakemake's documentation](https://snakemake.readthedocs.io/en/v6.1.1/).
+2. If you did not find an answer to your question, check [Snakemake's documentation](https://snakemake.readthedocs.io/en/v{{ snakemake_version }}/).
 3. If you still did not find an answer to your question, [open an issue](https://github.com/calliope-project/euro-calliope/issues/new/choose) on our issue tracker.
 
 ## Frequently Asked Questions

environment.yaml

@@ -6,5 +6,3 @@ dependencies:
     - python=3.11
     - pycountry=18.12.8
     - snakemake-minimal=8.10.7
-variables:
-    SNAKEMAKE_PROFILE: ./profiles/default

lib/eurocalliopelib/docs/dag.py

@@ -4,21 +4,19 @@
 
 import mkdocs
 import pydot
-import snakemake
 from mkdocs.plugins import BasePlugin
 from mkdocs.structure.files import File
+from snakemake import cli
 
 
 class DAGPlugin(BasePlugin):
     config_scheme = (
-        ("path_to_snakefile", mkdocs.config.config_options.Type(str)),
         ("path_to_src_dir", mkdocs.config.config_options.Type(str)),
         ("path_to_png_relative_to_site", mkdocs.config.config_options.Type(str)),
     )
 
     def on_files(self, files, config, **kwargs):
         """Generate DAG as png and add it to mkdocs files."""
-        path_to_snakemake = Path.cwd() / self.config["path_to_snakefile"]
         path_to_png = (
             Path.cwd()
             / self.config["path_to_src_dir"]
@@ -27,7 +25,8 @@ def on_files(self, files, config, **kwargs):
         path_to_png.parent.mkdir(parents=True, exist_ok=True)
         graph_string = io.StringIO()
         with redirect_stdout(graph_string):
-            snakemake.snakemake(path_to_snakemake, printrulegraph=True)
+            parser, args = cli.parse_args("--rulegraph")
+            cli.args_to_api(args, parser)
         graph = pydot.graph_from_dot_data(graph_string.getvalue())[0]
         graph.write_png(path_to_png)
 

mkdocs.yaml

@@ -30,8 +30,8 @@ markdown_extensions:
         permalink: "#"
 plugins:
     - search
+    - macros
     - dag:
-        path_to_snakefile: ./Snakefile
         path_to_src_dir: ./build/.docs
         path_to_png_relative_to_site: ./img/dag.png
     - schema:
@@ -43,3 +43,6 @@ plugins:
         path_to_src_dir: .
 
 extra_css: [./css/extras.css]
+extra:
+    snakemake_version: !!python/name:snakemake.__version__
+    calliope_version: 0.6.10

profiles/conda/config.yaml

@@ -1,4 +1,3 @@
-software-deployment-method: conda
+# These parameter are considered additional to ./profiles/default/config.yaml.
+# Apply them using the `--profile="profiles/conda` cli option.
 conda-frontend: conda
-cores: 2
-conda-prefix: ../envs/snakemake/euro-calliope/

profiles/default/config.yaml

@@ -1,3 +1,4 @@
 software-deployment-method: conda
+conda-frontend: mamba
 cores: 2
 conda-prefix: ../envs/snakemake/euro-calliope/

profiles/euler/config.yaml

@@ -1,10 +1,10 @@
+# These parameter are considered additional to ./profiles/default/config.yaml.
+# Apply them using the `--profile="profiles/euler` cli option.
 cluster: bsub -oo build/logs/{rule}.{wildcards}.log -W {resources.runtime} -n {resources.cores} -R "rusage[mem={resources.memory}]" -J {rule}.{wildcards} <
 jobs: 999
 local-cores: 1
 latency-wait: 60
 use-envmodules: True
-software-deployment-method: conda
-conda-prefix: ../envs/snakemake/euro-calliope/
 default-resources: [
     "runtime=10",
     "cores=1",

requirements-docs.yaml

@@ -3,14 +3,15 @@ channels:
     - conda-forge
     - bioconda
 dependencies:
-    - Python=3.8
+    - Python=3.11
     - mkdocs=1.2.2
+    - mkdocs-macros-plugin=1.0.5
     - jinja2=3.0.3
     - graphviz=2.48.0
     - pydot=1.4.2
     - pycountry=18.12.8
     - pymdown-extensions=9.9.2
-    - snakemake-minimal=7.26.0
+    - snakemake-minimal=8.10.7
     - pip=21.0.1
     - pip:
         - jsonschema2md==0.2.1