Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add subworkflows #479

Merged
merged 113 commits into from
Feb 11, 2022
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
113 commits
Select commit Hold shift + click to select a range
7fbfeb1
Add UMI subworkflow
FriederikeHanssen Jan 10, 2022
d95e34c
Get QUIASEQ tests to work + add CI test for UMI
FriederikeHanssen Jan 11, 2022
0758c11
Update gatk4 modules with newest version because of log4j
FriederikeHanssen Jan 12, 2022
7cd64c6
Update gatk4 modules with newest version because of log4j
FriederikeHanssen Jan 12, 2022
3b05722
Install nf-core subworklfows for tumor and joint_germline
FriederikeHanssen Jan 13, 2022
4865000
Add all the VC modules we need so far
FriederikeHanssen Jan 13, 2022
9c4cc53
Sort modules by usage in pipeline for readability
FriederikeHanssen Jan 13, 2022
6dd1192
Remove bam2fastq here again and port to own process and future subwor…
FriederikeHanssen Jan 13, 2022
347ecbc
Currently two separate bam2fq modules, before mapping collating needs…
FriederikeHanssen Jan 13, 2022
31c7fc9
Fix import
FriederikeHanssen Jan 13, 2022
fbb859b
Fix tests
FriederikeHanssen Jan 13, 2022
05dd3cd
First stab at bam2fastq subworkflow
FriederikeHanssen Jan 13, 2022
a4d50c5
Fix bam2fastq subworkflow
FriederikeHanssen Jan 14, 2022
eacda11
Fix md5sums
FriederikeHanssen Jan 14, 2022
65ecd54
Simple Bam2fastq with bam as pipeline input implementation, play arou…
FriederikeHanssen Jan 14, 2022
ba37c8a
sth
FriederikeHanssen Jan 14, 2022
8fb9ef9
add local deeptools module
FriederikeHanssen Jan 14, 2022
5546d88
add test names that need adding
FriederikeHanssen Jan 14, 2022
6150602
Use nf-core/deepvariant module
FriederikeHanssen Jan 17, 2022
39f7ca6
Update deepvariant module
FriederikeHanssen Jan 18, 2022
266940b
intervals=targets now, improve vc workflows
FriederikeHanssen Jan 19, 2022
8d716fb
account for some changed params
FriederikeHanssen Jan 19, 2022
e2ce1b2
Merge remote-tracking branch 'upstream/dev' into add_nfcore_subworkflows
FriederikeHanssen Jan 20, 2022
38d8a63
Expand germline VC to concat VCFs after slpitting by intervals
FriederikeHanssen Jan 20, 2022
2b7c006
Get step prepare_recal and test prepare_recal to work
FriederikeHanssen Jan 21, 2022
a564407
Add all the versioning I shamefully ignored until now
FriederikeHanssen Jan 21, 2022
d269197
Fix stalling of mapped reads
FriederikeHanssen Jan 21, 2022
d8607fd
use map and not groupKey for the remainder of the workflow
FriederikeHanssen Jan 21, 2022
1fc3191
Add default size 1 if no splitting is done
FriederikeHanssen Jan 22, 2022
849c3a3
get tests to pass
FriederikeHanssen Jan 22, 2022
96b4961
add/remove various modules
FriederikeHanssen Jan 22, 2022
631889b
check if it works w/o mqc
FriederikeHanssen Jan 22, 2022
87350a3
linting
FriederikeHanssen Jan 22, 2022
cd9fd11
import subworkflows again
FriederikeHanssen Jan 22, 2022
69479d5
import subworkflows again
FriederikeHanssen Jan 22, 2022
526fba6
uncomment failing tests
FriederikeHanssen Jan 22, 2022
3da275b
don't test for mqc atm
FriederikeHanssen Jan 22, 2022
78de2b2
skip intervals in bamqc right now
FriederikeHanssen Jan 23, 2022
4e7ac68
skip intervals in bamqc right now
FriederikeHanssen Jan 23, 2022
68d7cec
don't test qualimap right now
FriederikeHanssen Jan 24, 2022
7eb61f3
one test at a time
FriederikeHanssen Jan 24, 2022
c89c7aa
one test at a time
FriederikeHanssen Jan 24, 2022
9645806
one test at a time
FriederikeHanssen Jan 24, 2022
316e2a6
one test at a time
FriederikeHanssen Jan 24, 2022
71fe90e
one test at a time
FriederikeHanssen Jan 24, 2022
d2f7243
one test at a time
FriederikeHanssen Jan 24, 2022
5ec4203
add new md5sums for bam2fastq to account for new workflow
FriederikeHanssen Jan 24, 2022
1ccdc37
correct md5sum
FriederikeHanssen Jan 24, 2022
5d0602d
add splitfastq test
FriederikeHanssen Jan 24, 2022
a3c6f3d
add save_mapped test
FriederikeHanssen Jan 24, 2022
c690db3
add skip_md & prepare_recal
FriederikeHanssen Jan 24, 2022
b8b8f09
Change name
FriederikeHanssen Jan 24, 2022
8814b82
add tumor_normal_pair test
FriederikeHanssen Jan 24, 2022
8c96aa0
add last test
FriederikeHanssen Jan 24, 2022
2a2826f
add meta.lane to output name to avoid concurrentmodificatioexception …
FriederikeHanssen Jan 24, 2022
aa5535d
revert changes
FriederikeHanssen Jan 24, 2022
bcd9d47
Properly tumor/normal samples for VC
FriederikeHanssen Jan 25, 2022
582928c
skip targets for bamqc for the moment
FriederikeHanssen Jan 25, 2022
ddb75f8
tiddit needs upstream updating
FriederikeHanssen Jan 25, 2022
0454d9a
Remove tools tests until all paths are added
FriederikeHanssen Jan 25, 2022
f952ede
all the tests pass locally
FriederikeHanssen Jan 26, 2022
b389b1e
Remove test sam converter
FriederikeHanssen Jan 26, 2022
40cf342
Logic to get tumor only samples, @maxulysse this was a tough one
FriederikeHanssen Jan 27, 2022
f4af023
Add some tests with different combination of tumor-normal pairings, a…
FriederikeHanssen Jan 27, 2022
4f6e6d6
add comment
FriederikeHanssen Jan 27, 2022
3f749f1
add comment
FriederikeHanssen Jan 27, 2022
7ab8230
add no_intervals logic
FriederikeHanssen Feb 3, 2022
d8e2cd8
add wes intervals logic
FriederikeHanssen Feb 4, 2022
cbb2f91
make sure wes provides bed & qualimap gets complete bed file
FriederikeHanssen Feb 4, 2022
2eecefc
Properly merge scatter vcfs, adapt gatk4_tumor_only to do scatter/ga…
FriederikeHanssen Feb 4, 2022
1eb8f64
Call tumor vc again
FriederikeHanssen Feb 4, 2022
a83c2c0
Finish adaption of tumor_only gatk workflow
FriederikeHanssen Feb 4, 2022
cef8219
Add msisensorpro scan
FriederikeHanssen Feb 6, 2022
3174dc9
Add paired variant calling
FriederikeHanssen Feb 6, 2022
66bb5e0
Add paired variant calling
FriederikeHanssen Feb 6, 2022
27ea0a5
Add output folder
FriederikeHanssen Feb 6, 2022
47b8c96
Msisensor added for somatic samples
FriederikeHanssen Feb 7, 2022
b9fe16d
Fix nasty bug with combine and meta
FriederikeHanssen Feb 7, 2022
c30d0f6
simplify strelkaBP logic
FriederikeHanssen Feb 7, 2022
d6b384c
fix tests
FriederikeHanssen Feb 7, 2022
15acd37
fix prep genome for annotation
FriederikeHanssen Feb 7, 2022
bf365be
Merge remote-tracking branch 'upstream/dev' into add_nfcore_subworkflows
FriederikeHanssen Feb 8, 2022
38bcaae
Replace bed files, with bed files form test-data repo
FriederikeHanssen Feb 8, 2022
f7dce0f
add gitpod yml
FriederikeHanssen Feb 8, 2022
c7cd72a
Fix path to bed
FriederikeHanssen Feb 8, 2022
e997e90
fix new meta
FriederikeHanssen Feb 8, 2022
466f817
remove enabled:false, add mode:copy
FriederikeHanssen Feb 8, 2022
fe7399d
Use modules reads for umi testing
FriederikeHanssen Feb 8, 2022
d77f68e
update all the modules
FriederikeHanssen Feb 9, 2022
5ec3a22
update all the modules
FriederikeHanssen Feb 9, 2022
59ed939
Merge remote-tracking branch 'upstream/dev' into add_nfcore_subworkflows
FriederikeHanssen Feb 9, 2022
86f48d3
update msisensorpro
FriederikeHanssen Feb 9, 2022
9b33715
replace msisensor with nf-core
FriederikeHanssen Feb 9, 2022
7a07e63
add tumor/normal mutect2 wf
FriederikeHanssen Feb 9, 2022
97c228b
fix tests
FriederikeHanssen Feb 10, 2022
ed6ef9d
fix tests
FriederikeHanssen Feb 10, 2022
b43455a
update modules
FriederikeHanssen Feb 10, 2022
f141b42
Add pair mutect2 vc subworkflow
FriederikeHanssen Feb 10, 2022
c295e6d
update modules
FriederikeHanssen Feb 10, 2022
e491503
Update nextflow.config
FriederikeHanssen Feb 11, 2022
8d92354
add deeptools bamcoverage
FriederikeHanssen Feb 11, 2022
a2533f2
enable variant calling CI
FriederikeHanssen Feb 11, 2022
6dfd50b
update json with new qc option deeptools
FriederikeHanssen Feb 11, 2022
5c59775
comment subworkflows abck in
FriederikeHanssen Feb 11, 2022
893604c
publish bigwig file
FriederikeHanssen Feb 11, 2022
a718f0e
Update subworkflows/local/germline_variant_calling.nf
FriederikeHanssen Feb 11, 2022
6ecc7e0
add missing comma
FriederikeHanssen Feb 11, 2022
aed5f45
Thanks for the fix @nickhsmith
FriederikeHanssen Feb 11, 2022
9911a2e
Thanks for the fix @nickhsmith, fix typo in manta
FriederikeHanssen Feb 11, 2022
cb0829b
Convert interval to bed for correct handeling in varaint callers
FriederikeHanssen Feb 11, 2022
9dc8ba2
update targeted tests
FriederikeHanssen Feb 11, 2022
a30fc43
comment comment back in
FriederikeHanssen Feb 11, 2022
9ab5e61
Some simple check/error if somehting is not either a bed/interval file
FriederikeHanssen Feb 11, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ jobs:
- 'gatk4_spark'
#- 'save_bam_mapped'
- 'skip_markduplicates'
# - 'targeted'
- 'targeted'
- 'tumor_normal_pair'
- 'variant_calling'
steps:
Expand Down
38 changes: 38 additions & 0 deletions modules/local/gatk4/intervallisttobed.nf
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
process GATK4_INTERVALLISTTOBED {
tag "$meta.id"
label 'process_low'

conda (params.enable_conda ? "bioconda::gatk4=4.2.4.1" : null)
container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
'https://depot.galaxyproject.org/singularity/gatk4:4.2.4.1--hdfd78af_0':
'quay.io/biocontainers/gatk4:4.2.4.1--hdfd78af_0' }"

input:
tuple val(meta), path(interval)

output:
tuple val(meta), path("*.bed"), emit: bed
path "versions.yml" , emit: versions

script:
def args = task.ext.args ?: ''
def prefix = task.ext.prefix ?: "${meta.id}"

def avail_mem = 3
if (!task.memory) {
log.info '[GATK IntervalListToBed] Available memory not known - defaulting to 3GB. Specify process memory requirements to change this.'
} else {
avail_mem = task.memory.giga
}
"""
gatk --java-options "-Xmx${avail_mem}g" IntervalListToBed \\
--INPUT ${interval} \\
--OUTPUT ${meta.id}.bed \\
$args

cat <<-END_VERSIONS > versions.yml
"${task.process}":
gatk4: \$(echo \$(gatk --version 2>&1) | sed 's/^.*(GATK) v//; s/ .*\$//')
END_VERSIONS
"""
}
2 changes: 1 addition & 1 deletion subworkflows/local/germline_variant_calling.nf
Original file line number Diff line number Diff line change
Expand Up @@ -214,7 +214,7 @@ workflow GERMLINE_VARIANT_CALLING {
if(joint_germline){
run_haplotypecaller = false
run_vqsr = true //parameter?
//some feedback from gavin
some feedback from gavin
FriederikeHanssen marked this conversation as resolved.
Show resolved Hide resolved
// GATK_JOINT_GERMLINE_VARIANT_CALLING(
// haplotypecaller_vcf_gz_tbi,
// run_haplotypecaller,
Expand Down
7 changes: 7 additions & 0 deletions subworkflows/local/prepare_genome.nf
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ include { BWA_INDEX as BWAMEM1_INDEX } from '../../modu
include { BWAMEM2_INDEX } from '../../modules/nf-core/modules/bwamem2/index/main'
include { CREATE_INTERVALS_BED } from '../../modules/local/create_intervals_bed/main'
include { GATK4_CREATESEQUENCEDICTIONARY } from '../../modules/nf-core/modules/gatk4/createsequencedictionary/main'
include { GATK4_INTERVALLISTTOBED } from '../../modules/local/gatk4/intervallisttobed'
include { MSISENSORPRO_SCAN } from '../../modules/nf-core/modules/msisensorpro/scan/main'
include { SAMTOOLS_FAIDX } from '../../modules/nf-core/modules/samtools/faidx/main'
include { TABIX_BGZIPTABIX as TABIX_BGZIPTABIX_INTERVAL_SPLIT } from '../../modules/nf-core/modules/tabix/bgziptabix/main'
Expand Down Expand Up @@ -119,12 +120,18 @@ workflow PREPARE_GENOME {
}else{

tabix_in_combined = Channel.fromPath(file(params.intervals)).map{it -> [[id:it.baseName], it] }
if(!params.intervals.endsWith(".bed")){
GATK4_INTERVALLISTTOBED(tabix_in_combined)
tabix_in_combined = GATK4_INTERVALLISTTOBED.out.bed
ch_versions = ch_versions.mix(GATK4_INTERVALLISTTOBED.out.versions)
}
ch_intervals = CREATE_INTERVALS_BED(file(params.intervals))

}
}

if (!('annotate' in step) && !('controlfreec' in step)){

TABIX_BGZIPTABIX_INTERVAL_ALL(tabix_in_combined)
ch_intervals_combined_bed_gz_tbi = TABIX_BGZIPTABIX_INTERVAL_ALL.out.gz_tbi.map{ meta, bed, tbi -> [bed, tbi] }
ch_versions = ch_versions.mix(TABIX_BGZIPTABIX_INTERVAL_ALL.out.versions)
Expand Down