updating GATK #1419

asp8200 · 2024-02-22T14:58:33Z

Updating GATK to v4.5.

The test joint_germline fails with the new NF 24.1.0-edge. I don't think the failure is due to the upgrade of GATK, as we have seen it on the dev-branch before.

This PR also updates samtools in the GATK-markduplicates to v1.19.2. (That was a task left from #1412.) So after this PR has been merged, dev-Sarek will use samtools v1.19.2 everywhere.

github-actions · 2024-02-22T15:00:46Z

`nf-core lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit 9ab72f2

+| ✅ 182 tests passed       |+
#| ❔  13 tests were ignored |#
!| ❗   4 tests had warnings |!

❗ Test warnings:

pipeline_todos - TODO string in WorkflowSarek.groovy: Optionally add in-text citation tools to this list.
pipeline_todos - TODO string in main.nf: Optionally add in-text citation tools to this list.
pipeline_todos - TODO string in main.nf: Optionally add bibliographic entries to this list.
pipeline_todos - TODO string in main.nf: Only uncomment below if logic in toolCitationText/toolBibliographyText has been filled!

❔ Tests ignored:

files_exist - File is ignored: .github/workflows/awsfulltest.yml
files_exist - File is ignored: .github/workflows/awstest.yml
files_exist - File is ignored: conf/modules.config
files_exist - File is ignored: lib/WorkflowMain.groovy
files_exist - File is ignored: lib/NfcoreTemplate.groovy
files_exist - File is ignored: lib/WorkflowSarek.groovy
files_unchanged - File ignored due to lint config: assets/nf-core-sarek_logo_light.png
files_unchanged - File ignored due to lint config: docs/images/nf-core-sarek_logo_light.png
files_unchanged - File ignored due to lint config: docs/images/nf-core-sarek_logo_dark.png
files_unchanged - File ignored due to lint config: .gitignore or .prettierignore or pyproject.toml
actions_ci - actions_ci
actions_awstest - 'awstest.yml' workflow not found: /home/runner/work/sarek/sarek/.github/workflows/awstest.yml
template_strings - template_strings

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: assets/nf-core-sarek_logo_light.png
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/images/nf-core-sarek_logo_light.png
files_exist - File found: docs/images/nf-core-sarek_logo_dark.png
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: main.nf
files_exist - File found: assets/multiqc_config.yml
files_exist - File found: conf/base.config
files_exist - File found: conf/igenomes.config
files_exist - File found: modules.json
files_exist - File found: pyproject.toml
files_exist - File not found check: Singularity
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: pipeline_template.yml
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: docs/images/nf-core-sarek_logo.png
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: lib/Utils.groovy
files_exist - File not found check: lib/nfcore_external_java_deps.jar
files_exist - File not found check: .travis.yml
nextflow_config - Config variable found: manifest.name
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: manifest.homePage
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: params.validationShowHiddenParams
nextflow_config - Config variable found: params.validationSchemaIgnoreParams
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config manifest.name began with nf-core/
nextflow_config - Config variable manifest.homePage began with https://github.com/nf-core/
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config manifest.version ends in dev: 3.5dev
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
nextflow_config - nextflow.config contains configuration profile test
nextflow_config - Config default value correct: params.step= mapping
nextflow_config - Config default value correct: params.split_fastq= 50000000
nextflow_config - Config default value correct: params.nucleotides_per_second= 200000
nextflow_config - Config default value correct: params.clip_r1= 0
nextflow_config - Config default value correct: params.clip_r2= 0
nextflow_config - Config default value correct: params.three_prime_clip_r1= 0
nextflow_config - Config default value correct: params.three_prime_clip_r2= 0
nextflow_config - Config default value correct: params.trim_nextseq= 0
nextflow_config - Config default value correct: params.group_by_umi_strategy= Adjacency
nextflow_config - Config default value correct: params.aligner= bwa-mem
nextflow_config - Config default value correct: params.ascat_min_base_qual= 20
nextflow_config - Config default value correct: params.ascat_min_counts= 10
nextflow_config - Config default value correct: params.ascat_min_map_qual= 35
nextflow_config - Config default value correct: params.cf_coeff= 0.05
nextflow_config - Config default value correct: params.cf_contamination= 0
nextflow_config - Config default value correct: params.cf_minqual= 0
nextflow_config - Config default value correct: params.cf_mincov= 0
nextflow_config - Config default value correct: params.cf_ploidy= 2
nextflow_config - Config default value correct: params.sentieon_haplotyper_emit_mode= variant
nextflow_config - Config default value correct: params.sentieon_dnascope_emit_mode= variant
nextflow_config - Config default value correct: params.sentieon_dnascope_pcr_indel_model= CONSERVATIVE
nextflow_config - Config default value correct: params.dbnsfp_fields= rs_dbSNP,HGVSc_VEP,HGVSp_VEP,1000Gp3_EAS_AF,1000Gp3_AMR_AF,LRT_score,GERP++_RS,gnomAD_exomes_AF
nextflow_config - Config default value correct: params.vep_custom_args= --everything --filter_common --per_gene --total_length --offline --format vcf
nextflow_config - Config default value correct: params.vep_version= 110.0-0
nextflow_config - Config default value correct: params.vep_out_format= vcf
nextflow_config - Config default value correct: params.genome= GATK.GRCh38
nextflow_config - Config default value correct: params.igenomes_base= s3://ngi-igenomes/igenomes/
nextflow_config - Config default value correct: params.vep_cache= s3://annotation-cache/vep_cache/
nextflow_config - Config default value correct: params.snpeff_cache= s3://annotation-cache/snpeff_cache/
nextflow_config - Config default value correct: params.custom_config_version= master
nextflow_config - Config default value correct: params.custom_config_base= https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Config default value correct: params.test_data_base= https://raw.githubusercontent.com/nf-core/test-datasets/sarek3
nextflow_config - Config default value correct: params.seq_platform= ILLUMINA
nextflow_config - Config default value correct: params.max_cpus= 16
nextflow_config - Config default value correct: params.max_memory= 128.GB
nextflow_config - Config default value correct: params.max_time= 240.h
nextflow_config - Config default value correct: params.publish_dir_mode= copy
nextflow_config - Config default value correct: params.max_multiqc_email_size= 25.MB
nextflow_config - Config default value correct: params.validate_params= true
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - CODE_OF_CONDUCT.md matches the template
files_unchanged - LICENSE matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/CONTRIBUTING.md matches the template
files_unchanged - .github/ISSUE_TEMPLATE/bug_report.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/config.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/PULL_REQUEST_TEMPLATE.md matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - .github/workflows/linting.yml matches the template
files_unchanged - assets/email_template.html matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - assets/sendmail_template.txt matches the template
files_unchanged - docs/README.md matches the template
readme - README Nextflow minimum version badge matched config. Badge: 23.04.0, Config: 23.04.0
readme - README Zenodo placeholder was replaced with DOI.
pipeline_name_conventions - Name adheres to nf-core convention
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_lint - Input mimetype lint passed: 'text/csv'
schema_params - Schema matched params returned from nextflow config
system_exit - No System.exit calls found
actions_schema_validation - Workflow validation passed: branch.yml
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: cloudtest.yml
actions_schema_validation - Workflow validation passed: clean-up.yml
actions_schema_validation - Workflow validation passed: fix-linting.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
actions_schema_validation - Workflow validation passed: download_pipeline.yml
actions_schema_validation - Workflow validation passed: ncbench.yml
actions_schema_validation - Workflow validation passed: release-announcements.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
multiqc_config - 'assets/multiqc_config.yml' contains report_section_order
multiqc_config - 'assets/multiqc_config.yml' contains export_plots
multiqc_config - 'assets/multiqc_config.yml' contains report_comment
multiqc_config - 'assets/multiqc_config.yml' follows the ordering scheme of the minimally required plugins.
multiqc_config - 'assets/multiqc_config.yml' contains a matching 'report_comment'.
multiqc_config - 'assets/multiqc_config.yml' contains 'export_plots: true'.
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'

Run details

nf-core/tools version 2.13
Run at 2024-02-23 08:40:49

subworkflows/local/bam_variant_calling_haplotypecaller/main.nf

…HAPLOTYPECALLER

maxulysse · 2024-02-23T08:19:18Z

Looking amazing, just one comment left.

Regarding the joint germline failure, I can reproduce the pytest failure locally, but by running the code itself outside of pytest, it does work, so I'd say let's not waste time on this.

matthdsm · 2024-02-23T08:24:13Z

Looking amazing, just one comment left.

Regarding the joint germline failure, I can reproduce the pytest failure locally, but by running the code itself outside of pytest, it does work, so I'd say let's not waste time on this.

My experience is migrating from pytest > nf-test fixes a lot of issues with the testing. Perhaps we ca make it pass that way?

…HAPLOTYPECALLER

asp8200 · 2024-02-23T09:59:04Z

Can we merge this as-is and make a gitHub-issue for constructing an nf-test for joint-germline?

updating gatk/haplotypecaller

5cdc724

asp8200 added 2 commits February 22, 2024 15:10

Temporarily disabling nf-test for gatk/haplotypecaller

9694d25

Temporarily disabling nf-test for gatk4/haplotypecaller

1ec78f1

maxulysse reviewed Feb 22, 2024

View reviewed changes

subworkflows/local/bam_variant_calling_haplotypecaller/main.nf Outdated Show resolved Hide resolved

asp8200 added 25 commits February 22, 2024 15:31

updating gatk4/applybqsr and gatk4/applyvqsr

d320370

Removing redundant input-channel dbsnp_vqsr from BAM_VARIANT_CALLING_…

b995340

…HAPLOTYPECALLER

updating gatk4/baserecalibrator

fd62f6d

Temporarily disabling nf-test for gatk4/baserecalibrator

757f921

updating gatk4/calculatecontamination

8b2dbc2

updating gatk4/cnnscorevariants

be434c7

updating gatk4/createsequencedictionary

3efde7e

updating gatk4/estimatelibrarycomplexity

c12c40a

updating gatk4/filtermutectcalls

2b47d51

updating gatk4/filtervarianttranches

bec8c99

updating gatk4/gatherbqsrreports

50ad22c

updating gatk4/gatherpileupsummaries

b8c1902

updating gatk4/genomicsdbimport

dbcfb21

Temporarily disabling nf-test for gatk4/genomicsdbimport

057ef59

updating md5sum for mutect2.filtered.vcf.gz.filteringStats.tsv

1cf073a

updating gatk4/genotypegvcfs

d66027a

updating gatk4/getpileupsummaries

ee2f427

updating gatk4/intervallisttobed

2c15196

updating gatk4/learnreadorientationmodel

3ffb94c

updating gatk4/markduplicates

bd500cd

Temporarily disabling nf-test for gatk4/markduplicates

c0312f7

updating gatk4/mergemutectstats

f3fe6df

updating gatk4/mergevcfs

7cc1609

Temporarily disabling nf-test for gatk4/mergevcfs

dd135e9

updating gatk4/mutect2

b548bff

asp8200 added 2 commits February 22, 2024 19:19

updating gatk4/variantrecalibrator

1095957

Updating changelog

f998adb

asp8200 marked this pull request as ready for review February 22, 2024 20:35

asp8200 requested a review from FriederikeHanssen as a code owner February 22, 2024 20:35

Updating changelog

ac598b8

asp8200 added 2 commits February 23, 2024 08:38

Adding meta in certain input-channels in call to BAM_VARIANT_CALLING_…

9bb7bb4

…HAPLOTYPECALLER

Removing meta in certain input-channels in call to GATK4_HAPLOTYPECALLER

9ab72f2

maxulysse approved these changes Feb 23, 2024

View reviewed changes

FriederikeHanssen approved these changes Feb 23, 2024

View reviewed changes

maxulysse merged commit 3893bb5 into nf-core:dev Feb 23, 2024
55 of 57 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

updating GATK #1419

updating GATK #1419

asp8200 commented Feb 22, 2024 •

edited

Loading

github-actions bot commented Feb 22, 2024 •

edited

Loading

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

maxulysse commented Feb 23, 2024

matthdsm commented Feb 23, 2024

asp8200 commented Feb 23, 2024

updating GATK #1419

updating GATK #1419

Conversation

asp8200 commented Feb 22, 2024 • edited Loading

github-actions bot commented Feb 22, 2024 • edited Loading

nf-core lint overall result: Passed ✅ ⚠️

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

maxulysse commented Feb 23, 2024

matthdsm commented Feb 23, 2024

asp8200 commented Feb 23, 2024

asp8200 commented Feb 22, 2024 •

edited

Loading

github-actions bot commented Feb 22, 2024 •

edited

Loading

`nf-core lint` overall result: Passed ✅ ⚠️