Breaking changes in Snakemake 8

[jdblischak]

Snakemake 8.0.0 introduced some breaking changes. The most obvious that would need to be updated are:

• use-conda: True would need to be updated to software-deployment-method: conda
• use-conda: True + use-singularity: True would need to be updated to software-deployment-method: conda apptainer

But there are probably others. I don't have the time to dedicate to this at the moment, but I'm happy to review PRs from volunteers

[jdblischak]

Useful discussions to follow:

• Compatibility with snakemake version 8.0 and higher Snakemake-Profiles/slurm#117
• Interpolation of wildcards in profile slurm configuration snakemake/snakemake-executor-plugin-slurm#11

[JoshLoecker]

Introduction

I would like to start this process (initial work here), but I have a question about how to implement the changes required.

I'm going to be using README.md in the cluster-cancel folder for my example

The current header should be changed from # Cancel running jobs with `--cluster-cancel` to # Cancel running jobs with `--cluster-generic-submit-cmd` . It's obviously not great to remove all current documentation and update it to match snakemake>8.0 requirements, as not everyone may be on the latest version. Based on an assumption that the documentation should fit as many users' needs as possible, flags for both pre- and post- breaking changes should exist in the examples, which brings us to how these changes can be implemented

Options for Updates

Append documentation for snakemake-specific versions

It would be relatively easy to copy-paste documentation, changing flags where required. However, this would result in double the amount of documentation that is required to keep up-to-date in the future

# Snakemake <8.0
## Cancel running jobs with `--cluster-cancel`
(<8 docs here)

# Snakemake >8.0
## Cancel running jobs with `--cluster-generic-submit-cmd`
(>8 docs here)

Place snakemake>8-specific flags behind snakemake<8 flags

Include a paragraph in the repository's main README.md stating that flags for post-breaking changes are listed in parenthesis behind pre-breaking changes. This would reduce the amount of work required for future documentation updates. If this is done, an additional config.v8+.yaml file can be created for snakemake>8 users and can exist alongside the current config.yaml file in examples and the main simple/config.yaml for those using snakemake<8

# Cancel running jobs with `--cluster-cancel` (`--cluster-generic-submit-cmd`)
(continue docs here, no duplicate writing)

New installation requirements

In order to use the new cluster-generic-*-cmd arguments, the package snakemake-executor-plugin-cluster-generic must be installed using pip install snakemake-executor-plugin-cluster-generic. This can be included in the main conda.yml environment as follows

name: smk
channels:
  - conda-forge
  - bioconda
  - nodefaults
dependencies:
  - snakedeploy
  - snakefmt
  - snakemake>=5.15.0
  - pip
  
  - pip:
      - snakemake-executor-plugin-cluster-generic

Ending

If there are any other methods you would prefer or things you'd like to discuss, let me know!

[bheimbu]

Hi @jdblischak,

I was using your config.yaml for a while and everything was running smoothly, but now I've updated my snakemake installation (to 8.8.0) and it gives me several errors. I've uncommented reason: True and use-conda: True, however I get following error:

$ snakemake --profile profile/ -n
Using profile profile/ for setting default command line arguments.
Building DAG of jobs...
MissingRuleException:
No rule to produce --cluster=mkdir -p {OUTDIR}/logs/{rule} && sbatch --partition={resources.partition} --time={resources.time} --cpus-per-task={threads} --mem={resources.mem_mb} --job-name={rule}.{jobid} --output={OUTDIR}/logs/{rule}/{rule}_{wildcards}_%J.out --error={OUTDIR}/logs/{rule}/{rule}_{wildcards}_%J.err (if you use input functions make sure that they don't raise unexpected exceptions).

Everything did run smoothly under snakemake <8. I'm not sure how to change my config.yaml accordingly to use snakemake >8. Any help is highly appreciated.

Cheers Bastian

[jdblischak]

Everything did run smoothly under snakemake <8. I'm not sure how to change my config.yaml accordingly to use snakemake >8. Any help is highly appreciated.

@bheimbu Is there any specific reason you need to use snakemake >8? If not, you should downgrade to the latest version of snakemake 7, eg mamba install snakemake<8. This Issue is open in order to determine everything that needs to be changed. If you'd like to help contribute to this effort, your help would be appreciated. But if you just need your pipeline working again ASAP, the best solution is to continue using snakemake 7.

[jdblischak]

If there are any other methods you would prefer or things you'd like to discuss, let me know!

@JoshLoecker thanks so much for volunteering to help with this! It is super appreciated, and I apologize for my delayed response.

Ultimately I give you a lot of decision-making power since you are the one willing to work on it. I'm fine with your proposal of putting the new flag in parentheses and providing two config.yaml files. Though I will propose the alternative of having multiple branches. I can create the branches v7 and v8. v7 will preserve the v7 documentation, and v8 will contain your latest changes. Once you finish migrating everything to v8, then we can merge to main (while still including a link in the README to the v7 docs for those that need them). How does that sound? I think that could make it easier for you since you wouldn't have to maintain documentation and YAML files for multiple flags.

[jdblischak]

@JoshLoecker updates from my end: I created a branch v7 to preserve the snakemake <8 version of the documentation, pushed d6cbdbe to warn users in the README about the transition, and tagged v1.4.0 as the final release prior to migrating to snakemake 8.

[JoshLoecker]

That sounds great! I will update my fork to reflect this

[jdblischak]

Big thanks to @JoshLoecker for migrating the profile and docs to support Snakemake 8+! I just released v2.0.0

https://github.com/jdblischak/smk-simple-slurm/releases/tag/v2.0.0