Releases: nf-core/tools
Releases · nf-core/tools
v2.14.1 - Tantalum Toad - Patch
v2.14.0 - Tantalum Toad
This release contains some template changes and nf-core/tools updates.
🌟 Highlights 🌟
nf-core/tools functionalities
- We included a new linting test to ensure that nf-test snapshots contain the
versions.yml
file. - Components (modules and subworkflows) can now handle more possible git URLs (ssh:// and ftp://).
- The
nf-core download
command has a new argument--tag
, which allows adding additional tags to select particular revisions in the Seqera Platform interface. For example,--tag "3.10=validated"{:bash}
would allow you to quickly select the validated version of the pipeline.
Pipeline template
- We updated the GitHub Action which tests that the pipeline can be downloaded correctly (
download_pipeline.yml
) - We removed
pyproject.toml
from the template. - Pipeline-specific institutional configs support is now activated for all pipelines by default.
- The
.nf-core.yml
file contains now the version of the pipeline template.
Changelog
Template
- Remove fasta default from nextflow.config (#2828)
- Update templates to use nf-core/setup-nextflow v2 (#2818)
- Link to troubleshooting docs when pipeline fails (#2845)
- Add fallback to
download_pipeline.yml
in case the pipeline does not support stub runs (#2846) - Set topic variable correctly in the mastodon announcement (#2848)
- Add a cleanup action to
download_pipeline.yml
to fix failures caused by inadequate storage space on the runner (#2849) - Update python to 3.12 (#2805)
- Remove
pyproject.toml
from template root - Shorten lines in pipeline template (#2908)
- Add a new hidden
--pipelines_testdata_base_path
parameter to more easily switch locations of test data in test configs (#2931)[https://github.com//pull/2931] - Permanently activated pipeline-specific institutional configs support for all pipelines without need for manual intervention (#2936)
- Template config:
conda.channels
, notchannels
(#2950) - Handles multiple DOIs + doi.org resolver from manifest.doi (#2946)
- Update included components (#2949)
- Update .editorconfig (#2953)
Linting
- Only match assignments of params in
main.nf
and not references likeparams.aligner == <something>
(#2833) - Include test for presence of versions in snapshot (#2888)
- Components: set correct sha before running component lint tests (#2952)
- Less strict logo comparison (#2956)
- Handle request errors more gracefully for actions validation (#2959)
Download
- Replace
--tower
with--platform
. The former will remain for backwards compatability for now but will be removed in a future release. (#2853) - Better error message when GITHUB_TOKEN exists but is wrong/outdated
- New
--tag
argument to add custom tags during a pipeline download (#2938)
Components
- Handle more complete list of possible git URL forms (ssh:// and ftp:// prefixes specifically) (#2945)
- Fix path in component update script (#2823)
General
- Update CI to use nf-core/setup-nextflow v2 (#2819)
- Changelog bot: handle also patch version before dev suffix (#2820)
- Add
force_pr
flag to sync, to force a PR even though there are no changes committed (#2822) - Update prettier to 3.2.5 (#2830)
- Update GitHub Actions (#2827), (#2902), (#2927), (#2939)
- Switch to setup-nf-test (#2834)
- Add tests for assignment and referencing of params in main.nf (#2841)
- Optimize layers in dockerfile (#2842)
- Update python:3.11-slim Docker digest to a2eb07f (#2847)
- Strip out mention of "Nextflow Tower" and replace with "Seqera Platform" wherever possible
- Fix issue with config resolution that was causing nested configs to behave unexpectedly (#2862)
- Fix schema docs console output truncating (#2880)
- Ensure path object converted to string before stripping quotes (#2878)
- Fix incorrect assertions for called_with on mocks (#2891)
- Make cli-provided module/subworkflow names case insensitive (#2869)
- Get immediate parent path name for schema creation (#2886)
- Remove old references to CUSTOMDUMPSOFTWAREVERSIONS and add linting checks (#2897)
- Update pre-commit hook pre-commit/mirrors-mypy to v1.10.0 (#2933)
- Update codecov/codecov-action digest to 5ecb98a (#2948)
- Update gitpod/workspace-base Docker digest to 124f2b8 (#2943)
- fix(collectfile): sort true for methods_description_mqc.yaml (#2947)
- chore(deps): update pre-commit hook astral-sh/ruff-pre-commit to v0.4.3 (#2951)
- Restructure CHANGELOG.md (#2954)
v2.13.1 - Tin Puppy Patc
v2.13.1 - Tin Puppy Patch - [2024-02-29]
Patch release for failing AWS actions.
Template
- Remove obsolete editor settings in
devcontainer.json
andgitpod.yml
(#2795) - Add nf-test test instructions to contributing and PR template (#2807)
- Fix topic extraction step for hashtags in toots (#2810)
- Update modules and subworkflows in the template (#2811)
- Unpin setup-nextflow and action-tower-launch (#2806)
Download
- Improved offline container image resolution by introducing symlinks, fixes issues #2751, #2644 and demultiplex#164: (#2768)
General
v2.13 - Tin Puppy
v2.13 - Tin Puppy - [2024-02-20]
Highlights 🌟
This new version of nf-core/tools includes a refactoring of the pipelines template.
- The
lib
directory is removed: 👋 Groovy code - We now use nf-core subworkflows for pipeline initialisation (replacing the code from the
lib
directory) - The
nf-validation
plugin is now used to create an input channel from a sample sheet file.
Check the nf-core blog post for more information!
Template
- Add empty line in README.md to fix badges. (#2729)
- Replace automatic branch detection in
nf-core download
CI test with hardcodeddev
and input. (#2727) - Add Github Action to automatically cleanup ubuntu-latest runners to fix runner running out of diskspace errors(#2755)
- Fix GitHub Actions CI and Linting badges links (#2757)
- Add hashtags to release announcement on mastodon (#2761)
- update fastqc and multiqc in template (#2776)
- template refactoring: remove the
lib
directory and use nf-core subworkflows (#2736) - use nf-validation to create an input channel from a sample sheet (#2736)
Linting
- Make creat-lint-wf composable (#2733)
- Add looser comparison when pipeline logos (#2744)
- Handle multiple aliases in module imports correctly during linting (#2762)
- Switch to markdown based API and error docs (#2758)
Modules
- Handle dirty local module repos by force checkout of commits and branches if needed (#2734)
- Patch: handle file not found when it is an added file to a module (#2771)
- Handle symlinks when migrating pytest (#2770)
- Add
--profile
parameter to nf-test command (#2767)
General
- fix ignoring changes in partially templated files (e.g.
.gitignore
) (#2722) - update ruff to 0.2.0 and add it to pre-commit step (#2725)
- Update codecov/codecov-action digest to e0b68c6 (#2728)
- Update pre-commit hook astral-sh/ruff-pre-commit to v0.2.1 (#2730)
- Update python:3.11-slim Docker digest to 2a746e2 (#2743)
- Update actions/setup-python action to v5 (#2739)
- Update gitpod/workspace-base Docker digest to 45e7617 (#2747)
- chore(deps): pin jlumbroso/free-disk-space action to 54081f1 (#2756)
- chore(deps): update actions/github-script action to v7 (#2766)
- chore(deps): update pre-commit hook astral-sh/ruff-pre-commit to v0.2.2 (#2769)
- Update gitpod/workspace-base Docker digest to 728e1fa (#2780)
v2.12.1 - Aluminium Wolf - Patch
v2.12.1 - Aluminium Wolf - Patch - [2024-02-01]
Linting
- Handle default values of type number from nextflow schema (#2703)
- fix ignoring files_unchanged (#2707)
General
- Update pre-commit hook astral-sh/ruff-pre-commit to v0.1.15 (#2705)
- use types for default value comparison (#2712)
- fix changelog titles (#2708)
- Print relative path not absolute path in logo cmd log output (#2709)
- Update codecov/codecov-action action to v4 (#2713)
- Ignore nf-core-bot in renovate PRs (#2716)
2.12 - Aluminium Wolf
v2.12 - Aluminium Wolf - [2024-01-29]
Template
- Add a Github Action Workflow to the pipeline template that tests a successful download with 'nf-core download' (#2618)
- Use
pre-commit
to lint files in GitHub CI (#2635) - Use pdiff also on gitpod for nf-test (#2640)
- switch to new image syntax in readme (#2645)
- Add conda channel order to nextflow.config (#2094)
- Fix tyop in pipeline nextflow.config (#2664)
- Remove
nfcore_external_java_deps.jar
from lib directory in pipeline template (#2675) - Add function to check
-profile
is well formatted (#2678) - Add new pipeline error message pointing to docs when 'requirement exceeds available memory' error message (#2680)
- add 👀👍🏻🎉😕 reactions to fix-linting-bot action (#2692)
Linting
- Fix linting of a pipeline with patched custom module (#2669)
- linting a pipeline also lints the installed subworkflows (#2677)
- environment.yml name must be lowercase (#2676)
- allow ignoring specific files when template_strings (#2686)
- lint
nextflow.config
default values match the ones specified innextflow_schema.json
(#2684)
Modules
- Fix empty json output for
nf-core list local
(#2668)
General
- Run CI-pytests for nf-core tools on self-hosted runners (#2550)
- Add Ruff linter and formatter replacing Black, isort and pyupgrade (#2620)
- Set pdiff as nf-test differ in Docker image for Gitpod (#2642)
- Fix Renovate Dockerfile updating issues (#2648 and #2651)
- Add new subcommand
nf-core tui
, which launches a TUI (terminal user interface) to intuitively explore the command line flags, built using Trogon (#2655) - Add new subcommand:
nf-core logo-create
to output an nf-core logo for a pipeline (instead of going through the website) (#2662) - Handle api redirects from the old site (#2672)
- Remove redundanct v in pipeline version for emails (#2667)
- add function to check
-profile
is well formatted (#2678) - Update pre-commit hook astral-sh/ruff-pre-commit to v0.1.14 (#2674)
- Update pre-commit hook pre-commit/mirrors-mypy to v1.8.0 (#2630)
- Update mshick/add-pr-comment action to v2 (#2632)
- update python image version in docker file (#2636)
- Update actions/cache action to v4 (#2666)
- Update peter-evans/create-or-update-comment action to v4 (#2683) and (#2695)
Magnesium Dragon Patch
v2.11.1 - Magnesium Dragon Patch - [2023-12-20]
Template
- Rename
release-announcments.yml
torelease-announcements.yml
(#2610) - Fix
nextflow.config
docker.runOptions
(#2607)
General
- Only dump
modules.json
when it is modified (#2609)
2.11
v2.11 - Magnesium Dragon + [2023-12-19]
Highlights:
- nf-test is now used for testing modules and subworkflows. 🚀 This comes with updated templates and a changed commands (good-bye
nf-core modules/subworkflows create-test-yml
). For more information, you can refer to the revised documentation on modules and subworkflows .- Every module and subworkflow now contains a
tests/
directory, which includes amain.nf.test
file. By runningnf-core modules test
ornf-core subworkflows test
, you can perform the tests and generate themain.nf.test.snap
snapshot file. As a general guideline, we recommend testing all outputs of modules/subworkflows. For complex cases, please refer to the nf-test assertion tutorial. - To assist with the conversion of modules and subworkflows from pytest to nf-test, we've added the new
--migrate-pytest
option fornf-core modules test
andnf-core subworkflows test
, which does most of the work for you. - The
nf-core modules lint
andnf-core subworkflows lint
commands now also check the nf-test configuration and warn you if you are using pytest (it's really not that hard to switch 🙂 ).
- Every module and subworkflow now contains a
- We pinned the version of nf-validation, which should make offline usage of nf-core pipelines easier.
- In
nf-core download
, we now prefix all absolute container URIs withdocker://
, which should make more pipelines work out of the box. - We started using type hinting in the nf-core/tools codebase. This will help us to find bugs earlier and make the code more readable.
The complete list of changes can be found below.
Template
- Fix writing files to a remote outdir in the NfcoreTemplate helper functions (#2465)
- Fancier syntax highlighting for example samplesheets in the usage.md template (#2503)
- Use closure for multiqc ext.args (#2509)
- Fix how the modules template references the conda environment file (#2540)
- Unset env variable JAVA_TOOL_OPTIONS in gitpod (#2569)
- Pin the version of nf-validation (#2579)
- Disable process selector warnings by default (#2161)
- Remove
docker.userEmulation
from nextflow.config in pipeline template (#2580)
Download
- Add
docker://
prefix for absolute container URIs as well (#2576). - Bugfix for AttributeError:
ContainerError
object has no attributeabsoluteURI
(#2543).
Linting
- Fix incorrectly failing linting if 'modules' was not found in meta.yml (#2447)
- Correctly pass subworkflow linting test if
COMPONENT.out.versions
is used in the script (#2448) - Add pyupgrade to pre-commit config and dev requirements as mentioned in #2200
- Check for spaces in modules container URLs (#2452)
- Correctly ignore
timeline.enabled
,report.enabled
,trace.enabled
,dag.enabled
variables when linting a pipeline. (#2507) - Lint nf-test main.nf.test tags include all used components in chained tests (#2572)
- Don't fail linting if md5sum for empty files are found in a stub test (#2571)
- Check for existence of test profile (#2478)
Modules
- Added stub test creation to
create_test_yml
(#2476) - Replace ModulePatch by ComponentPatch (#2482)
- Fixed
nf-core modules lint
to work with new module structure for nf-test (#2494) - Add option
--migrate-pytest
to create a module with nf-test taking into account an existing module (#2549) - When installing modules and subworkflows, automatically create the
./modules
directory if it doesn't exist (#2563) - When
.nf-core.yml
is not found create it in the current directory instead of the root filesystem (#2237) - Modules
--migrate-pytest
copies template scripts (#2568)
Subworkflows
- Added stub test creation to
create_test_yml
(#2476) - Fixed
nf-core subworkflows lint
to work with new module structure for nf-test (#2494) - Add option
--migrate-pytest
to create a subworkflow with nf-test taking into account an existing subworkflow (#2549)
General
- Update
schema build
functionality to automatically update defaults which have changed in thenextflow.config
(#2479) - Change testing framework for modules and subworkflows from pytest to nf-test (#2490)
bump_version
keeps now the indentation level of the updated version entries (#2514)- Add mypy to pre-commit config for the tools repo (#2545)
- Use Path objects for ComponentCreate and update the structure of components templates (#2551).
- GitPod base image: swap tool installation back to
conda
frommamba
(#2566). - Sort the
installed_by
list inmodules.json
(#2570). - Unset env variable JAVA_TOOL_OPTIONS in gitpod (#2569)
New Contributors
- @fa2k made their first contribution in #2472
- @NovakApis made their first contribution in #2223
- @rernst made their first contribution in #2507
- @kedhammar made their first contribution in #2483
v2.10 - Nickel Ostrich
v2.10 - Nickel Ostrich + [2023-09-25]
This new release includes new commands, template updates, and several bug fixes. Thanks to all contributors for making this possible!
🌟 Highlights
- We are now linting subworkflows
nf-core subworkflows lint
. - Use
nf-core create-params-file
params-file with documentation and default values for all parameters in a pipeline. - A new GitHub Action will automatically post new pipeline release messages to Mastodon, Bluesky and Twitter
- Pipelines in sync with the new template, will dump all pipeline parameters to a file called
params.json
after every run
✖️ Deprecations
- Deprecation of the
—no-git
argument fornf-core create
command - Deprecation of the
—template-yaml
argument for thenf-core sync
command - The same information will now be automatically stored in.nf-core.yml
Template
- Fix links in
multiqc_config.yml
(#2372 and #2412) - Remove default false from nextflow_schema.json (#2376)
- Add module MULTIQC to modules.config (#2377)
- Add GitHub workflow for automated release announcements (#2382)
- Update the Code of Conduct (#2381)
- Save template information to
.nf-core.yml
and deprecate argument--template-yaml
fornf-core sync
(#2388 and #2389) - (#2397) Remove fixed Ubuntu test and added to standard testing matrix
- (#2396) Reduce container finding error to warning since the registries are not consistent.
- (#2415) Add autoMounts for apptainer.
- Remove
igenomes_base
from the schema, so that nf-validation doesn't create a file path and throw errors offline for s3 objects. - Modified devcontainer permissions so that singularity can be run in Codespaces/VS Code devcontainers (Commit a103f44)
- Update Gitpod profile resources to reflect base environment settings.
- (#747) Add to the template the code to dump the selected pipeline parameters into a json file.
Download
- Improved container image resolution and prioritization of http downloads over Docker URIs (#2364).
- Registries provided with
-l
/--container-library
will be ignored for modules with explicit container registry specifications (#2403). - Fix unintentional downloading of containers in test for the Tower download functionality. Bug reported by @adamrtalbot and @awgymer (#2434).
Linting
- Add new command
nf-core subworkflows lint
(#2379)
Modules
Subworkflows
- Fix bug: missing subworkflow name when using
nf-core subworkflows create
(#2435)
General
- Initialise
docker_image_name
to fixUnboundLocalError
error (#2374) - Fix prompt pipeline revision during launch (#2375)
- Add a
create-params-file
command to create a YAML parameter file for a pipeline containing parameter documentation and defaults. (#2362) - Update the Code of Conduct (#2381)
- Remove
--no-git
option fromnf-core create
(#2394) - Throw warning when custom workflow name contains special characters (#2401)
- Bump version of nf-test snapshot files with
nf-core bump-version
(#2410)
v2.9 - Chromium Falcon
v2.9 - Chromium Falcon + [2023-06-30]
A new big release with new features, template updates and bug fixes. Thanks to all contributors!
🌟 Highlights
- A new
nf-core download
command compatible with Nextflow Tower. - Implementation of the plugin
nf-validation
for parameter validation in pipeline templates. - Nextflow version bump to 23.04.0.
Template
params.max_multiqc_email_size
is no longer required (#2273)- Remove
cleanup = true
fromtest_full.config
in pipeline template (#2279) - Fix usage docs for specifying
params.yaml
(#2279) - Added stub in modules template ([#2277])(#2277) [Contributed by @nvnieuwk]
- Move registry definitions out of profile scope ([#2286])(#2286)
- Remove
aws_tower
profile ([#2287])(#2287) - Fixed the Slack report to include the pipeline name (#2291)
- Fix link in the MultiQC report to point to exact version of output docs (#2298)
- Updates seqeralabs/action-tower-launch to v2.0.0 (#2301)
- Remove schema validation from
lib
folder and use Nextflow nf-validation plugin instead (#1771) - Fix parsing of container directive when it is not typical nf-core format (#2306)
- Add ability to specify custom registry for linting modules, defaults to quay.io (#2313)
- Add
singularity.registry = 'quay.io'
in pipeline template (#2305) - Add
apptainer.registry = 'quay.io'
in pipeline template (#2352) - Bump minimum required NF version in pipeline template from
22.10.1
->23.04.0
(#2305) - Add ability to interpret
docker.registry
fromnextflow.config
file. If not found defaults to quay.io. (#2318) - Add functions to dynamically include pipeline tool citations in MultiQC methods description section for better reporting. (#2326)
- Remove
--tracedir
parameter (#2290) - Incorrect config parameter warnings when customising pipeline template (#2333)
- Use markdown syntax in the description for the meta map channels (#2358)
Download
- Introduce a
--tower
flag fornf-core download
to obtain pipelines in an offline format suited for seqeralabs® Nextflow Tower (#2247). - Refactored the CLI for
--singularity-cache
innf-core download
from a flag to an argument. The prior options were renamed toamend
(container images are only saved in the$NXF_SINGULARITY_CACHEDIR
) andcopy
(a copy of the image is saved with the download).remote
was newly introduced and allows to provide a table of contents of a remote cache via an additional argument--singularity-cache-index
(#2247). - Refactored the CLI parameters related to container images. Although downloading other images than those of the Singularity/Apptainer container system is not supported for the time being, a generic name for the parameters seemed preferable. So the new parameter
--singularity-cache-index
introduced in #2247 has been renamed to--container-cache-index
prior to release (#2336). - To address issue #2311, a new parameter
--container-library
was created allowing to specify the container library (registry) from which container images in OCI format (Docker) should be pulled (#2336). - Container detection in configs was improved. This allows for DSL2-like container definitions inside the container parameter value provided to process scopes #2346.
- Add apptainer to the list of false positve container strings (#2353).
Updated CLI parameters
Old parameter | New parameter |
---|---|
new parameter | -d / --download-configuration |
new parameter | -t / --tower |
-c / --container |
-s / --container-system <VALUE> |
new parameter | -l / --container-library <VALUE> |
--singularity-cache |
-u / --container-cache-utilisation <VALUE> |
new parameter | -i / --container-cache-index <VALUE> |
In addition, -r
/ --revision
has been changed to a parameter that can be provided multiple times so several revisions can be downloaded at once.
Linting
- Warn if container access is denied (#2270)
- Error if module container specification has quay.io as prefix when it shouldn't have ([#2278])(https://github.com/nf-core/tools/pull/2278/files)
- Detect if container is 'simple name' and try to contact quay.io server by default (#2281)
- Warn about null/None/empty default values in
nextflow_schema.json
(#3328) - Fix linting when creating a pipeline skipping some parts of the template and add CI test (#2330)
Modules
- Don't update
modules_json
object if a module is not updated (#2323)
Subworkflows
General
- GitPod base image: Always self-update to the latest version of Nextflow. Add pre-commit dependency.
- GitPod configs: Update Nextflow as an init task, init pre-commit in pipeline config.
- Refgenie: Create
nxf_home/nf-core/refgenie_genomes.config
path if it doesn't exist (#2312) - Add CI tests to test running a pipeline when it's created from a template skipping different areas