Update to DSL2 Best Practices #379

maxulysse · 2021-05-20T11:44:31Z

PR checklist

github-actions · 2021-05-20T11:48:04Z

`nf-core lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit 68e1cc4

+| ✅ 134 tests passed       |+
#| ❔  11 tests were ignored |#
!| ❗  82 tests had warnings |!

❗ Test warnings:

files_exist - File not found: environment.yml
files_exist - File not found: Dockerfile
nextflow_config - Config variable not found: process.container
params_used - Config variable not found in main.nf: params.input
params_used - Config variable not found in main.nf: params.step
params_used - Config variable not found in main.nf: params.genome
params_used - Config variable not found in main.nf: params.genomes_base
params_used - Config variable not found in main.nf: params.save_reference
params_used - Config variable not found in main.nf: params.help
params_used - Config variable not found in main.nf: params.no_intervals
params_used - Config variable not found in main.nf: params.nucleotides_per_second
params_used - Config variable not found in main.nf: params.sentieon
params_used - Config variable not found in main.nf: params.skip_qc
params_used - Config variable not found in main.nf: params.target_bed
params_used - Config variable not found in main.nf: params.tools
params_used - Config variable not found in main.nf: params.trim_fastq
params_used - Config variable not found in main.nf: params.clip_r1
params_used - Config variable not found in main.nf: params.clip_r2
params_used - Config variable not found in main.nf: params.three_prime_clip_r1
params_used - Config variable not found in main.nf: params.three_prime_clip_r2
params_used - Config variable not found in main.nf: params.trim_nextseq
params_used - Config variable not found in main.nf: params.save_trimmed
params_used - Config variable not found in main.nf: params.split_fastq
params_used - Config variable not found in main.nf: params.aligner
params_used - Config variable not found in main.nf: params.markdup_java_options
params_used - Config variable not found in main.nf: params.use_gatk_spark
params_used - Config variable not found in main.nf: params.save_bam_mapped
params_used - Config variable not found in main.nf: params.skip_markduplicates
params_used - Config variable not found in main.nf: params.ascat_ploidy
params_used - Config variable not found in main.nf: params.ascat_purity
params_used - Config variable not found in main.nf: params.cf_coeff
params_used - Config variable not found in main.nf: params.cf_contamination
params_used - Config variable not found in main.nf: params.cf_contamination_adjustment
params_used - Config variable not found in main.nf: params.cf_ploidy
params_used - Config variable not found in main.nf: params.cf_window
params_used - Config variable not found in main.nf: params.generate_gvcf
params_used - Config variable not found in main.nf: params.no_strelka_bp
params_used - Config variable not found in main.nf: params.pon
params_used - Config variable not found in main.nf: params.pon_index
params_used - Config variable not found in main.nf: params.ignore_soft_clipped_bases
params_used - Config variable not found in main.nf: params.umi
params_used - Config variable not found in main.nf: params.read_structure1
params_used - Config variable not found in main.nf: params.read_structure2
params_used - Config variable not found in main.nf: params.annotate_tools
params_used - Config variable not found in main.nf: params.annotation_cache
params_used - Config variable not found in main.nf: params.cadd_cache
params_used - Config variable not found in main.nf: params.cadd_indels
params_used - Config variable not found in main.nf: params.cadd_indels_tbi
params_used - Config variable not found in main.nf: params.cadd_wg_snvs
params_used - Config variable not found in main.nf: params.cadd_wg_snvs_tbi
params_used - Config variable not found in main.nf: params.genesplicer
params_used - Config variable not found in main.nf: params.snpeff_cache
params_used - Config variable not found in main.nf: params.config_profile_contact
params_used - Config variable not found in main.nf: params.config_profile_description
params_used - Config variable not found in main.nf: params.config_profile_url
params_used - Config variable not found in main.nf: params.outdir
params_used - Config variable not found in main.nf: params.publish_dir_mode
params_used - Config variable not found in main.nf: params.sequencing_center
params_used - Config variable not found in main.nf: params.multiqc_config
params_used - Config variable not found in main.nf: params.monochrome_logs
params_used - Config variable not found in main.nf: params.email
params_used - Config variable not found in main.nf: params.email_on_fail
params_used - Config variable not found in main.nf: params.plaintext_email
params_used - Config variable not found in main.nf: params.max_multiqc_email_size
params_used - Config variable not found in main.nf: params.hostnames
params_used - Config variable not found in main.nf: params.validate_params
params_used - Config variable not found in main.nf: params.tracedir
params_used - Config variable not found in main.nf: params.enable_conda
params_used - Config variable not found in main.nf: params.pull_docker_container
params_used - Config variable not found in main.nf: params.cpus
params_used - Config variable not found in main.nf: params.max_cpus
params_used - Config variable not found in main.nf: params.max_memory
params_used - Config variable not found in main.nf: params.max_time
actions_awsfulltest - .github/workflows/awsfulltest.yml should test full datasets, not -profile test
readme - README did not have a Nextflow minimum version badge.
pipeline_todos - TODO string in main.nf: It MUST be possible to pass additional parameters to the tool as a command-line string via the "$ioptions.args" variable
pipeline_todos - TODO string in main.nf: If the tool supports multi-threading then you MUST provide the appropriate parameter
pipeline_todos - TODO string in test_full.config: Specify the paths to your full test data ( on nf-core/test-datasets or directly in repositories, e.g. SRA)
pipeline_todos - TODO string in test_full.config: Give any required params for the test so that command line flags are not needed
pipeline_todos - TODO string in awsfulltest.yml: You can customise AWS full pipeline tests as required
pipeline_todos - TODO string in awstest.yml: You can customise CI pipeline run tests as required
schema_description - No description provided in schema for parameter: cpus

❔ Tests ignored:

files_unchanged - File ignored due to lint config: .github/ISSUE_TEMPLATE/bug_report.md
files_unchanged - File ignored due to lint config: .github/ISSUE_TEMPLATE/feature_request.md
files_unchanged - File ignored due to lint config: .github/PULL_REQUEST_TEMPLATE.md
files_unchanged - File ignored due to lint config: assets/nf-core-sarek_logo.png
files_unchanged - File ignored due to lint config: docs/images/nf-core-sarek_logo.png
files_unchanged - File ignored due to lint config: lib/NfcoreSchema.groovy
files_unchanged - File ignored due to lint config: .gitignore or foo
files_unchanged - File does not exist: .github/workflows/push_dockerhub_dev.yml
files_unchanged - File does not exist: .github/workflows/push_dockerhub_release.yml
conda_env_yaml - No environment.yml file found - skipping conda_env_yaml test
conda_dockerfile - No environment.yml / Dockerfile file found - skipping conda_dockerfile test

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: CHANGELOG.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File found: .github/markdownlint.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: assets/nf-core-sarek_logo.png
files_exist - File found: bin/markdown_to_html.py
files_exist - File found: docs/images/nf-core-sarek_logo.png
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: lib/nfcore_external_java_deps.jar
files_exist - File found: lib/NfcoreSchema.groovy
files_exist - File found: main.nf
files_exist - File found: conf/base.config
files_exist - File found: .github/workflows/awstest.yml
files_exist - File found: .github/workflows/awsfulltest.yml
files_exist - File not found check: Singularity
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .travis.yml
nextflow_config - Config variable found: manifest.name
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: manifest.homePage
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: params.show_hidden_params
nextflow_config - Config variable found: params.schema_ignore_params
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable (correctly) not found: params.version
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config manifest.name began with nf-core/
nextflow_config - Config variable manifest.homePage began with https://github.com/nf-core/
nextflow_config - Config dag.file ended with .svg
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config manifest.version ends in dev: '3.0dev'
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
params_used - Config variable found in main.nf: params.vep_cache
params_used - Config variable found in main.nf: params.ac_loci
params_used - Config variable found in main.nf: params.ac_loci_gc
params_used - Config variable found in main.nf: params.bwa
params_used - Config variable found in main.nf: params.chr_dir
params_used - Config variable found in main.nf: params.chr_length
params_used - Config variable found in main.nf: params.dbsnp
params_used - Config variable found in main.nf: params.dbsnp_index
params_used - Config variable found in main.nf: params.dict
params_used - Config variable found in main.nf: params.fasta
params_used - Config variable found in main.nf: params.fasta_fai
params_used - Config variable found in main.nf: params.germline_resource
params_used - Config variable found in main.nf: params.germline_resource_index
params_used - Config variable found in main.nf: params.intervals
params_used - Config variable found in main.nf: params.known_indels
params_used - Config variable found in main.nf: params.known_indels_index
params_used - Config variable found in main.nf: params.mappability
params_used - Config variable found in main.nf: params.snpeff_db
params_used - Config variable found in main.nf: params.species
params_used - Config variable found in main.nf: params.vep_cache_version
files_unchanged - .gitattributes matches the template
files_unchanged - CODE_OF_CONDUCT.md matches the template
files_unchanged - LICENSE matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/CONTRIBUTING.md matches the template
files_unchanged - .github/ISSUE_TEMPLATE/config.yml matches the template
files_unchanged - .github/markdownlint.yml matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - .github/workflows/linting.yml matches the template
files_unchanged - assets/email_template.html matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - assets/sendmail_template.txt matches the template
files_unchanged - bin/markdown_to_html.py matches the template
files_unchanged - docs/README.md matches the template
files_unchanged - lib/nfcore_external_java_deps.jar matches the template
files_unchanged - assets/multiqc_config.yaml matches the template
actions_ci - '.github/workflows/ci.yml' is triggered on expected events
actions_ci - '.github/workflows/ci.yml' checks minimum NF version
actions_awstest - '.github/workflows/awstest.yml' is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml is triggered correctly
pipeline_name_conventions - Name adheres to nf-core convention
template_strings - Did not find any Jinja template strings (217 files)
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_params - Schema matched params returned from nextflow config
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: awsfulltest.yml
actions_schema_validation - Workflow validation passed: awstest.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
actions_schema_validation - Workflow validation passed: local_modules.yml
actions_schema_validation - Workflow validation passed: branch.yml
merge_markers - No merge markers found in pipeline files

Run details

nf-core/tools version 1.14
Run at 2021-06-08 08:37:54

maxulysse · 2021-05-20T12:06:48Z

Following the @drpatelh DSL2 Best Pratices

…erything

maxulysse · 2021-05-30T09:34:29Z

OK, so now back on track, with some tiny added improvements:
With nf-core/test-datasets#262, I introduced samplesheet.csv files in sarek
With nf-core/test-datasets#263, I want to try out how minimal such a file can be within sarek, and how flexible we can be with our csv files.

I really love the samplesheet check process introduced by @drpatelh in the DSL2 pipeline, but I feel that we're still wasting a process for that, which is what I'm exploring here in Sarek:

sarek/workflows/sarek.nf

Lines 376 to 412 in 4fbbf80

    
           def extract_csv(csv_file) { 
        
               Channel.from(csv_file) 
        
                   .splitCsv(header: true) 
        
                   .map{ row -> 
        
                       def meta = [:] 
        
                       meta.patient = row.patient 
        
                       meta.sample  = row.sample.toString() 
        
                       // If no gender specified, gender is not considered (only used for somatic CNV) 
        
                       if (row.gender == null) { 
        
                           meta.gender = "NA" 
        
                       } else meta.gender = row.gender.toString() 
        
                       // If no status specified, sample is considered normal 
        
                       if (row.status == null) { 
        
                           meta.status = 0 
        
                       } else meta.status = row.status.toInteger() 
        
                       if (row.lane == null) { 
        
                       // variant_calling 
        
                           meta.id = meta.sample 
        
                           def bam     = file(row.bam, checkIfExists: true) 
        
                           def bai     = file(row.bai, checkIfExists: true) 
        
                           return [meta, bam, bai] 
        
                       } else { 
        
                       // mapping with fastq 
        
                           meta.id         = "${row.sample}-${row.lane}".toString() 
        
                           def read1       = file(row.fastq1, checkIfExists: true) 
        
                           def read2       = file(row.fastq2, checkIfExists: true) 
        
                           def CN          = params.sequencing_center ? "CN:${params.sequencing_center}\\t" : '' 
        
                           def read_group  = "\"@RG\\tID:${row.lane}\\t${CN}PU:${row.lane}\\tSM:${row.sample}\\tLB:${row.sample}\\tPL:ILLUMINA\"" 
        
                           meta.read_group = read_group 
        
                           return [meta, [read1, read2]] 
        
                       } 
        
                   } 
        
           }

Plan is to asses as much as we can from the header and infer the rest.
So won't change much for our regular users (apart from switching to DSL2 and having tons of images instead of one, and switching from a tsv to a csv file with a header).
But it could simplify access for new users and we don't need all the information all the time.
We only need gender when doing cnv, and we only need status for tumor/normal pairs.
So if it's not specified, why not assigning by default a 0 as status and "NA" as gender?
Can we remove anything else?
Do we need any more information?

@nf-core/core @nf-core/sarek what do you think?

maxulysse · 2021-06-07T07:30:04Z

ok, now that I fixed CI once again, I'll now update all current modules.
And then extend CI

FriederikeHanssen

looks good. This is the PR depending on something weird with the test data, right?

conf/test.config

main.nf

modules/nf-core/software/samtools/stats/main.nf

maxulysse · 2021-06-11T13:35:52Z

looks good. This is the PR depending on something weird with the test data, right?

Just with CSV instead of TSV actually.
(but yeah now we can go weird with the CSV too).

maxulysse added 2 commits May 20, 2021 13:43

feat: update main script and lib from DSL2 Best Practices

1e0edf8

fix: add more file to lint ignore

b120e90

maxulysse added 21 commits May 20, 2021 14:11

feat: strip out unused functions

b5fc96c

feat: update some modules

34a0b4d

fix: HaplotypeCaller

14d5b97

feat: update msisensor

bcc8385

feat: update msisensor_msi

f68fa78

fix: Gstring.toString()

b86fc4f

fix: output

7f55703

remove unused file

7be61c8

remove white space

6291ec5

fix: output

8f6e461

fix: markduplicates subworkflow test

ce7b5ee

fix: subworkflow md finally fixed

4141f64

fix: nextflow version

99dd4ea

moving files and functions around

b1d14e0

feat: update modules and move around functions, and possibly break ev…

8db1c6f

…erything

fix: add .gitignore to .nf-core-lint.yml

151d600

fix: add new csv files

712e29d

feat: simplify mapping script, update module, default test working

b83e31f

feat: cleanup files

ea0b43e

cleanup

266772b

feat: simplify variant_calling

c371418

maxulysse mentioned this pull request May 28, 2021

feat: simplify csv files nf-core/test-datasets#263

Merged

maxulysse added 3 commits May 30, 2021 10:27

fix: output

6c3299d

feat: simplify tests

77a5c65

fix: tests

4fbbf80

maxulysse added 20 commits June 7, 2021 10:37

feat: update bwa/bwamem2 modules

4332455

feat: update trimgalore

2b3b55c

feat: update tabix/tabix

7ed1a92

feat: update and add strelka modules

7b2a7a3

fix: typo

61eee21

fix: path to module

b214c5d

fix: typo

0f1fba5

feat: update samtools/merge

04d97f4

feat: update multiqc modules

b4a903c

feat: replace msisensor by msisensor-pro

7dc1c86

fix: path to modules

8d10fc4

fix: msisensor-pro test

54bd1d4

fix: msisensorpro execution7

7d4547b

feat: code polishing

28b232b

fix: improve csv files generation

214a28b

feat: code polishing

6b23e8e

fix: update CI

f52c318

fix: add csv for recalibrated bam files

dacc326

feat: restart from all steps

b092ea3

fix: gatk4spark tests

0dd07ea

maxulysse marked this pull request as ready for review June 8, 2021 08:12

feat: code polishing

68e1cc4

maxulysse requested review from FriederikeHanssen and drpatelh June 8, 2021 19:43

maxulysse mentioned this pull request Jun 8, 2021

Dsl2 annotation #386

Merged

11 tasks

FriederikeHanssen approved these changes Jun 11, 2021

View reviewed changes

conf/test.config Show resolved Hide resolved

main.nf Show resolved Hide resolved

modules/nf-core/software/samtools/stats/main.nf Show resolved Hide resolved

maxulysse merged commit 39fd254 into nf-core:dsl2 Jun 11, 2021

maxulysse deleted the dsl2_modules_update branch June 11, 2021 13:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update to DSL2 Best Practices #379

Update to DSL2 Best Practices #379

maxulysse commented May 20, 2021

github-actions bot commented May 20, 2021 •

edited

Loading

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

maxulysse commented May 20, 2021

maxulysse commented May 30, 2021

maxulysse commented Jun 7, 2021 •

edited

Loading

FriederikeHanssen left a comment

maxulysse commented Jun 11, 2021

Update to DSL2 Best Practices #379

Update to DSL2 Best Practices #379

Conversation

maxulysse commented May 20, 2021

PR checklist

github-actions bot commented May 20, 2021 • edited Loading

nf-core lint overall result: Passed ✅ ⚠️

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

maxulysse commented May 20, 2021

maxulysse commented May 30, 2021

maxulysse commented Jun 7, 2021 • edited Loading

FriederikeHanssen left a comment

Choose a reason for hiding this comment

maxulysse commented Jun 11, 2021

github-actions bot commented May 20, 2021 •

edited

Loading

`nf-core lint` overall result: Passed ✅ ⚠️

maxulysse commented Jun 7, 2021 •

edited

Loading