[DRAFT] Add NGSCheckMate in as part of a cram sampleQC subworkflow #1252

SPPearce · 2023-09-24T20:24:58Z

This PR is to add NGSCheckMate, a tool to check if samples come from the same individual.
I have put it into a subworkflow (currently called cram_sampleqc), on the thought that we will include sexdeterrmine and somalier at the same point.

github-actions · 2023-09-24T20:27:24Z

`nf-core lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit 4e55d5d

+| ✅ 144 tests passed       |+
#| ❔   9 tests were ignored |#
!| ❗   4 tests had warnings |!

❗ Test warnings:

pipeline_todos - TODO string in WorkflowSarek.groovy: Optionally add in-text citation tools to this list.
schema_description - No description provided in schema for parameter: bcftools_annotations
schema_description - No description provided in schema for parameter: bcftools_annotations_index
schema_description - No description provided in schema for parameter: bcftools_header_lines

❔ Tests ignored:

files_exist - File is ignored: .github/workflows/awsfulltest.yml
files_exist - File is ignored: conf/modules.config
files_unchanged - File ignored due to lint config: assets/nf-core-sarek_logo_light.png
files_unchanged - File ignored due to lint config: docs/images/nf-core-sarek_logo_light.png
files_unchanged - File ignored due to lint config: docs/images/nf-core-sarek_logo_dark.png
files_unchanged - File ignored due to lint config: lib/NfcoreTemplate.groovy
files_unchanged - File ignored due to lint config: .gitignore or .prettierignore or pyproject.toml
actions_ci - actions_ci
template_strings - template_strings

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: assets/nf-core-sarek_logo_light.png
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/images/nf-core-sarek_logo_light.png
files_exist - File found: docs/images/nf-core-sarek_logo_dark.png
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: lib/nfcore_external_java_deps.jar
files_exist - File found: lib/NfcoreTemplate.groovy
files_exist - File found: lib/Utils.groovy
files_exist - File found: lib/WorkflowMain.groovy
files_exist - File found: main.nf
files_exist - File found: assets/multiqc_config.yml
files_exist - File found: conf/base.config
files_exist - File found: conf/igenomes.config
files_exist - File found: .github/workflows/awstest.yml
files_exist - File found: lib/WorkflowSarek.groovy
files_exist - File found: modules.json
files_exist - File found: pyproject.toml
files_exist - File not found check: Singularity
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: pipeline_template.yml
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: docs/images/nf-core-sarek_logo.png
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: .travis.yml
nextflow_config - Config variable found: manifest.name
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: manifest.homePage
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: params.validationShowHiddenParams
nextflow_config - Config variable found: params.validationSchemaIgnoreParams
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config manifest.name began with nf-core/
nextflow_config - Config variable manifest.homePage began with https://github.com/nf-core/
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config manifest.version ends in dev: 3.4dev
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - CODE_OF_CONDUCT.md matches the template
files_unchanged - LICENSE matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/CONTRIBUTING.md matches the template
files_unchanged - .github/ISSUE_TEMPLATE/bug_report.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/config.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/PULL_REQUEST_TEMPLATE.md matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - .github/workflows/linting.yml matches the template
files_unchanged - assets/email_template.html matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - assets/sendmail_template.txt matches the template
files_unchanged - docs/README.md matches the template
files_unchanged - lib/nfcore_external_java_deps.jar matches the template
actions_awstest - '.github/workflows/awstest.yml' is triggered correctly
readme - README Nextflow minimum version badge matched config. Badge: 23.04.0, Config: 23.04.0
readme - README Zenodo placeholder was replaced with DOI.
pipeline_name_conventions - Name adheres to nf-core convention
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_lint - Input mimetype lint passed: 'text/csv'
schema_params - Schema matched params returned from nextflow config
system_exit - No System.exit calls found
actions_schema_validation - Workflow validation passed: branch.yml
actions_schema_validation - Workflow validation passed: release-announcments.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: awstest.yml
actions_schema_validation - Workflow validation passed: clean-up.yml
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: fix-linting.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
multiqc_config - 'assets/multiqc_config.yml' follows the ordering scheme of the minimally required plugins.
multiqc_config - 'assets/multiqc_config.yml' contains a matching 'report_comment'.
multiqc_config - 'assets/multiqc_config.yml' contains 'export_plots: true'.
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'

Run details

nf-core/tools version 2.10
Run at 2023-11-07 22:52:29

conf/igenomes.config

maxulysse · 2023-09-28T16:14:49Z

conf/modules/ngscheckmate.config

@@ -0,0 +1,12 @@
+process {
+    withName: ".*BAM_NGSCHECKMATE:BCFTOOLS_MPILEUP" {
+        ext.when   = { params.tools && params.tools.split(',').contains('sampleqc') }


why sampleqc and not ngscheckmate?

I was thinking that the other tools that have been discussed (sexdeterrmine, somalier) would go into the same subworkflow. So I thought they might be all controlled together, but if they should be more granular that is fine too.

I'd rather be granular, as we already have some qc tools

agreed. maybe someone wants to run checkmate but not somalier etc.

To be fair, for most our QC tools, they enabled by default and disabled à la demande with --skip_tools, should we do that there?

CHANGELOG.md

maxulysse

Looking amazing, can you add some tests maybe?

conf/modules/ngscheckmate.config

SPPearce · 2023-09-29T04:34:01Z

Looking amazing, can you add some tests maybe?

Yes, I do want to add a test, just wasn't sure where/how to add that so need to discuss please.

conf/igenomes.config

CHANGELOG.md

Co-authored-by: Maxime U Garcia <max.u.garcia@gmail.com>

…ngscheckmate

Co-authored-by: Maxime U Garcia <max.u.garcia@gmail.com>

FriederikeHanssen

Looks good, just some smaller additional things with doc ordering from me. I am blaming @maxulysse for training me for 3 years 😆

CHANGELOG.md

docs/output.md

nextflow.config

github-actions · 2023-10-11T08:31:56Z

Python linting (`black`) is failing

To keep the code consistent with lots of contributors, we run automated code consistency checks.
To fix this CI test, please run:

Install black: pip install black
Fix formatting errors in your pipeline: black .

Once you push these changes the test should pass, and you can hide this comment 👍

We highly recommend setting up Black in your code editor so that this formatting is done automatically on save. Ask about it on Slack for help!

Thanks again for your contribution!

SPPearce · 2023-11-07T08:59:53Z

@maxulysse , I can't seem to manage to specify ngscheckmate_bed using the test_data, can you help me please.
Swapping
ngscheckmate_bed = params.test_data['homo_sapiens']['genome']['ngscheckmate_bed']
instead of hardcoding the path (in conf/test/cache.config and conf/test/test.config) gives me an error:

ERROR ~ Unknown config attribute params.ngscheckmate_bed -- check config file: /workspace/sarek/nextflow.config

SPPearce · 2023-11-07T09:06:30Z

@maxulysse , I can't seem to manage to specify ngscheckmate_bed using the test_data, can you help me please. Swapping ngscheckmate_bed = params.test_data['homo_sapiens']['genome']['ngscheckmate_bed'] instead of hardcoding the path (in conf/test/cache.config and conf/test/test.config) gives me an error:

ERROR ~ Unknown config attribute params.ngscheckmate_bed -- check config file: /workspace/sarek/nextflow.config

Ah, I think I need this: nf-core/modules#4308

conf/test.config

nextflow.config

maxulysse · 2023-11-07T16:11:48Z

need something there: https://github.com/nf-core/sarek/blob/master/main.nf#L54

Co-authored-by: Maxime U Garcia <max.u.garcia@gmail.com>

conf/test.config

subworkflows/nf-core/bam_ngscheckmate/main.nf

FriederikeHanssen

looks really great, the only thing it is missing from is the full size tests, and the workflow overview images

Co-authored-by: Friederike Hanssen <Friederike.hanssen@qbic.uni-tuebingen.de>

SPPearce · 2023-11-07T16:33:52Z

looks really great, the only thing it is missing from is the full size tests, and the workflow overview images

Yes, what full size test should I put it into?

maxulysse · 2023-11-07T17:53:41Z

looks really great, the only thing it is missing from is the full size tests, and the workflow overview images

Yes, what full size test should I put it into?

both

SPPearce · 2023-11-07T18:21:06Z

looks really great, the only thing it is missing from is the full size tests, and the workflow overview images

Yes, what full size test should I put it into?

both

Ok, to be more specific, where are the full size tests specified?

FriederikeHanssen · 2023-11-07T22:03:11Z

the failing tests are due to changes in the sentieon structure. mergeing dev in should fix it

SPPearce added 2 commits September 24, 2023 20:21

Add NGSCheckMate

94ca07d

Update CHANGELOG

21d38a2

SPPearce requested a review from maxulysse as a code owner September 24, 2023 20:24

SPPearce changed the base branch from master to dev September 24, 2023 20:25

nf-core deleted a comment from github-actions bot Sep 24, 2023

Update tools list for sampleqc

2e72d1c

maxulysse reviewed Sep 28, 2023

View reviewed changes

conf/igenomes.config Outdated Show resolved Hide resolved

maxulysse reviewed Sep 28, 2023

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

maxulysse reviewed Sep 28, 2023

View reviewed changes

conf/modules/ngscheckmate.config Show resolved Hide resolved

Update CHANGELOG.md

cee8614

Merge branch 'dev' into ngscheckmate

9c48c87

maxulysse reviewed Sep 29, 2023

View reviewed changes

conf/igenomes.config Outdated Show resolved Hide resolved

maxulysse reviewed Sep 29, 2023

View reviewed changes

conf/igenomes.config Outdated Show resolved Hide resolved

maxulysse reviewed Sep 29, 2023

View reviewed changes

conf/igenomes.config Outdated Show resolved Hide resolved

maxulysse reviewed Sep 29, 2023

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

SPPearce and others added 11 commits September 29, 2023 20:40

Update conf/igenomes.config

a2496de

Co-authored-by: Maxime U Garcia <max.u.garcia@gmail.com>

Update conf/igenomes.config

05f681e

Co-authored-by: Maxime U Garcia <max.u.garcia@gmail.com>

Update CHANGELOG.md

9d9105c

Co-authored-by: Maxime U Garcia <max.u.garcia@gmail.com>

Update conf/igenomes.config

2ab1e4d

Co-authored-by: Maxime U Garcia <max.u.garcia@gmail.com>

Update conf/igenomes.config

ac649e0

Co-authored-by: Maxime U Garcia <max.u.garcia@gmail.com>

Add tests, swap to ngscheckmate

a0e5f43

Merge branch 'dev' into ngscheckmate

2c95984

Fix NGSCheckMate test name

97e1fa6

Merge remote-tracking branch 'refs/remotes/origin/ngscheckmate' into …

64273f3

…ngscheckmate

Update test

ad97096

Update output path and docs

7378fe9

Update tests/config/tags.yml

4e73bdb

Co-authored-by: Maxime U Garcia <max.u.garcia@gmail.com>

FriederikeHanssen reviewed Oct 10, 2023

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

docs/output.md Outdated Show resolved Hide resolved

nextflow.config Outdated Show resolved Hide resolved

Merge branch 'dev' into ngscheckmate

8f59abc

SPPearce added 3 commits November 7, 2023 08:28

Fix merge conflict

3acaef9

Apply code review suggestions, fix channel to mpileup

e1706a3

Swap around bed location in confs

01ae112

Swap to modules test-data

13df981

maxulysse reviewed Nov 7, 2023

View reviewed changes

conf/test.config Show resolved Hide resolved

maxulysse reviewed Nov 7, 2023

View reviewed changes

nextflow.config Outdated Show resolved Hide resolved

Apply suggestions from code review

32227d3

Co-authored-by: Maxime U Garcia <max.u.garcia@gmail.com>

FriederikeHanssen reviewed Nov 7, 2023

View reviewed changes

conf/test.config Outdated Show resolved Hide resolved

FriederikeHanssen reviewed Nov 7, 2023

View reviewed changes

subworkflows/nf-core/bam_ngscheckmate/main.nf Show resolved Hide resolved

FriederikeHanssen reviewed Nov 7, 2023

View reviewed changes

SPPearce and others added 3 commits November 7, 2023 16:24

Add getGenomeAttribute check

10185fa

Update conf/test.config

ac87102

Co-authored-by: Friederike Hanssen <Friederike.hanssen@qbic.uni-tuebingen.de>

Move blank line

cf1e116

SPPearce added 2 commits November 7, 2023 22:50

Add to somatic full test

5f25439

Merge remote-tracking branch 'origin/dev' into ngscheckmate

4e55d5d

FriederikeHanssen approved these changes Nov 7, 2023

View reviewed changes

maxulysse approved these changes Nov 8, 2023

View reviewed changes

FriederikeHanssen merged commit 966919b into dev Nov 8, 2023
25 checks passed

maxulysse deleted the ngscheckmate branch January 17, 2024 09:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DRAFT] Add NGSCheckMate in as part of a cram sampleQC subworkflow #1252

[DRAFT] Add NGSCheckMate in as part of a cram sampleQC subworkflow #1252

SPPearce commented Sep 24, 2023 •

edited

Loading

github-actions bot commented Sep 24, 2023 •

edited

Loading

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

maxulysse Sep 28, 2023

SPPearce Sep 29, 2023

maxulysse Sep 29, 2023

FriederikeHanssen Sep 29, 2023

maxulysse Sep 29, 2023

maxulysse left a comment

SPPearce commented Sep 29, 2023

FriederikeHanssen left a comment

github-actions bot commented Oct 11, 2023

SPPearce commented Nov 7, 2023

SPPearce commented Nov 7, 2023

maxulysse commented Nov 7, 2023

FriederikeHanssen left a comment

SPPearce commented Nov 7, 2023

maxulysse commented Nov 7, 2023

SPPearce commented Nov 7, 2023

FriederikeHanssen commented Nov 7, 2023

[DRAFT] Add NGSCheckMate in as part of a cram sampleQC subworkflow #1252

[DRAFT] Add NGSCheckMate in as part of a cram sampleQC subworkflow #1252

Conversation

SPPearce commented Sep 24, 2023 • edited Loading

github-actions bot commented Sep 24, 2023 • edited Loading

nf-core lint overall result: Passed ✅ ⚠️

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

maxulysse Sep 28, 2023

Choose a reason for hiding this comment

SPPearce Sep 29, 2023

Choose a reason for hiding this comment

maxulysse Sep 29, 2023

Choose a reason for hiding this comment

FriederikeHanssen Sep 29, 2023

Choose a reason for hiding this comment

maxulysse Sep 29, 2023

Choose a reason for hiding this comment

maxulysse left a comment

Choose a reason for hiding this comment

SPPearce commented Sep 29, 2023

FriederikeHanssen left a comment

Choose a reason for hiding this comment

github-actions bot commented Oct 11, 2023

Python linting (black) is failing

SPPearce commented Nov 7, 2023

SPPearce commented Nov 7, 2023

maxulysse commented Nov 7, 2023

FriederikeHanssen left a comment

Choose a reason for hiding this comment

SPPearce commented Nov 7, 2023

maxulysse commented Nov 7, 2023

SPPearce commented Nov 7, 2023

FriederikeHanssen commented Nov 7, 2023

SPPearce commented Sep 24, 2023 •

edited

Loading

github-actions bot commented Sep 24, 2023 •

edited

Loading

`nf-core lint` overall result: Passed ✅ ⚠️

Python linting (`black`) is failing