Skip to content

Commit

Permalink
Update default profile to be a workflow-profile
Browse files Browse the repository at this point in the history
Default profile (in `profiles/default`) is now the default workflow profile. Snakemake activates it automatically using the name. This allows the user to inject other profile parameters using the `--profile` cli option. This also allows us to provide the conda configuration as an addition to, rather than a replacement of, the default profile.

Fixes calliope-project#268
  • Loading branch information
timtroendle committed Apr 15, 2024
1 parent 4fa9cd0 commit 2dc0987
Show file tree
Hide file tree
Showing 6 changed files with 8 additions and 17 deletions.
4 changes: 2 additions & 2 deletions docs/workflow/build-remote.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Build on a cluster

You may want to build the model on a cluster.
While you can build Euro-Calliope on [any cluster that is supported by Snakemake](https://snakemake.readthedocs.io/en/v6.1.1/executing/cluster.html), our default configuration is targeted at, and tested on, ETH's Euler cluster.
While you can build Euro-Calliope on [any cluster that is supported by Snakemake](https://snakemake.readthedocs.io/en/v8.10.7/tutorial/additional_features.html#cluster-or-cloud-execution), our default configuration is targeted at, and tested on, ETH's Euler cluster.

## Build

Expand All @@ -11,7 +11,7 @@ To build the model on Euler, use the following command:
snakemake --profile profiles/euler
```

If you want to run on another cluster, read [snakemake's documentation on cluster execution](https://snakemake.readthedocs.io/en/stable/executing/cluster.html) and take `profiles/euler` as a starting point.
If you want to run on another cluster, read [snakemake's documentation on cluster execution](https://snakemake.readthedocs.io/en/v8.10.7/tutorial/additional_features.html#cluster-or-cloud-execution) and take `profiles/euler` as a starting point.

## Work local, build on remote

Expand Down
10 changes: 2 additions & 8 deletions docs/workflow/customisation.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,19 +13,13 @@ You can, for example, change the temporal and spatial scope of the data and mode

The configuration builds on Snakemake's configuration mechanism and consists of two parts: a default configuration `./config/default.yaml` and a schema declaring all configuration parameters `./config/schema.yaml`.
To override configuration parameters, you can add another configuration file with just your updates or change parameter values on the command line when calling `snakemake`.
For details on how the configuration mechanism works, please read [Snakemake's documention](https://snakemake.readthedocs.io/en/v6.1.1/snakefiles/configuration.html).

!!! warning
When you change configuration parameters, please consider this important caveat of Snakemake:
While Snakemake does understand that parameters have changed, it will not rerun rules with updated parameters by default (see also this [feature request](https://github.com/snakemake/snakemake/issues/976)).
To ensure that your configuration parameters are used, you must manually rerun all rules that use the updated parameter.
The easiest is to start the build process from scratch by running `snakemake clean`.
For details on how the configuration mechanism works, please read [Snakemake's documention](https://snakemake.readthedocs.io/en/v8.10.7/snakefiles/configuration.html).

## Adaptation

Beyond configuration through parameters, you can adapt and extend the workflow in any possible way.
You can adapt the data pre-processing steps and the way model files are generated, but you can also extend the model by adding your own model files or overrides.
Customising Euro-Calliope in this way requires a solid understanding of the workflow management system [Snakemake](https://snakemake.readthedocs.io/en/v6.1.1/index.html) that we use.
Customising Euro-Calliope in this way requires a solid understanding of the workflow management system [Snakemake](https://snakemake.readthedocs.io/en/v8.10.7/index.html) that we use.

Whenever we applied Euro-Calliope in our research we made use of this option.
Below you will find a list of publications in which we applied Euro-Calliope models.
Expand Down
2 changes: 1 addition & 1 deletion docs/workflow/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

Euro-Calliope's workflow builds the models from raw data.
In essence, it is composed out of three things: scripts that retrieve raw data, process data, and formulate the model; execution environments that explicitly define software dependencies and requirements (equivalent to the conditions in a lab); and the glue ensuring all scripts run in the correct execution environment and the right order.
Scripts are written in Python; the execution environments are [conda environments](https://docs.conda.io/en/latest/); and the glue holding everything together is the workflow management system [Snakemake](https://snakemake.readthedocs.io/en/v6.1.1/).
Scripts are written in Python; the execution environments are [conda environments](https://docs.conda.io/en/latest/); and the glue holding everything together is the workflow management system [Snakemake](https://snakemake.readthedocs.io/en/v8.10.7/).

## Rationale

Expand Down
2 changes: 1 addition & 1 deletion docs/workflow/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ running into these problems is completely normal and in many cases a solution is
Here are some steps we advise you to go through:

1. Read through our frequently asked questions below.
2. If you did not find an answer to your question, check [Snakemake's documentation](https://snakemake.readthedocs.io/en/v6.1.1/).
2. If you did not find an answer to your question, check [Snakemake's documentation](https://snakemake.readthedocs.io/en/v8.10.7/).
3. If you still did not find an answer to your question, [open an issue](https://github.com/calliope-project/euro-calliope/issues/new/choose) on our issue tracker.

## Frequently Asked Questions
Expand Down
2 changes: 0 additions & 2 deletions environment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6,5 +6,3 @@ dependencies:
- python=3.11
- pycountry=18.12.8
- snakemake-minimal=8.10.7
variables:
SNAKEMAKE_PROFILE: ./profiles/default
5 changes: 2 additions & 3 deletions profiles/conda/config.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
software-deployment-method: conda
# These parameter are considered additional to ./profiles/default/config.yaml.
# Apply them using the `--profile="profiles/conda` cli option.
conda-frontend: conda
cores: 2
conda-prefix: ../envs/snakemake/euro-calliope/

0 comments on commit 2dc0987

Please sign in to comment.