Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update modules once again #459

Merged
merged 22 commits into from
Dec 2, 2021
Merged

Update modules once again #459

merged 22 commits into from
Dec 2, 2021

Conversation

maxulysse
Copy link
Member

PR checklist

  • This comment contains a description of changes (with reason).
  • If you've fixed a bug or added code that should be tested, add tests!
    • If you've added a new tool - add to the software_versions process and a regex to scrape_software_versions.py
    • If you've added a new tool - have you followed the pipeline conventions in the contribution docs
    • If necessary, also make a PR on the nf-core/sarek branch on the nf-core/test-datasets repository.
  • Make sure your code lints (nf-core lint .).
  • Ensure the test suite passes (nextflow run . -profile test,docker).
  • Usage Documentation in docs/usage.md is updated.
  • Output Documentation in docs/output.md is updated.
  • CHANGELOG.md is updated.
  • README.md is updated (including new tool citations and authors/contributors).

@github-actions
Copy link

github-actions bot commented Nov 29, 2021 •
edited
Loading

nf-core lint overall result: Passed ✅ ⚠️

Posted for pipeline commit 18209c0

+| ✅ 133 tests passed       |+
#| ❔   5 tests were ignored |#
!| ❗   5 tests had warnings |!

❗ Test warnings:

  • pipeline_todos - TODO string in awsfulltest.yml: You can customise AWS full pipeline tests as required
  • pipeline_todos - TODO string in test_full.config: Specify the paths to your full test data ( on nf-core/test-datasets or directly in repositories, e.g. SRA)
  • pipeline_todos - TODO string in test_full.config: Give any required params for the test so that command line flags are not needed
  • pipeline_todos - TODO string in test_full.config: Specify the paths to your full test data ( on nf-core/test-datasets or directly in repositories, e.g. SRA)
  • pipeline_todos - TODO string in test_full.config: Give any required params for the test so that command line flags are not needed

❔ Tests ignored:

  • files_unchanged - File ignored due to lint config: .github/workflows/linting.yml
  • files_unchanged - File ignored due to lint config: assets/nf-core-sarek_logo.png
  • files_unchanged - File ignored due to lint config: docs/images/nf-core-sarek_logo.png
  • files_unchanged - File ignored due to lint config: lib/NfcoreTemplate.groovy
  • files_unchanged - File ignored due to lint config: assets/multiqc_config.yaml

✅ Tests passed:

Run details

  • nf-core/tools version 2.2.dev0
  • Run at 2021-12-01 21:49:56

@maxulysse maxulysse marked this pull request as ready for review December 1, 2021 21:33
@maxulysse
Copy link
Member Author

@drpatelh at first I was a big annoyed because I just had finally finished my refactoring of everything with the newest modules syntax a week ago, and I had to do it again with the newer newest syntax.
But the simplification is so good, and I had a lot of fun playing with it.
It definitively will allow us to port a lot of subworkflows into the nf-core/modules repo.
It's even a bit scary how much you can control the processes within the modules.config.

I'm particularly proud of the rework of our local bwamem and bwamem2 modules that are now stripped of every sarek logic, and the rework it allowed for the mapping subworkflow.
<3
I'm really looking forward a tools release now.

FriederikeHanssen
FriederikeHanssen reviewed Dec 2, 2021
View reviewed changes
conf/modules.config
// MAPPING
process {

withName: "BWA.*MEM" {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

😍

FriederikeHanssen
FriederikeHanssen reviewed Dec 2, 2021
View reviewed changes
conf/modules.config
// Only name sort if Spark for Markduplicates + duplicate marking is not skipped
if (('markduplicates' in params.use_gatk_spark) && (!params.skip_markduplicates)) {
process {
withName: "BWA.*_MEM" { ext.args2 = 'sort -n' }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This then overwrites the above config ext.args2 for BWAMEM?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm hoping that we can also remove the part thingy from the bwa*mem processes

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will think about how to do this. I am sure there is a way

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where are you using it? I don't see it in the process anymore

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh yes, that's what I wanted to figure it out.
I'll let someone else try that, I'm going to the NGI retreat today, so won't be able to work much

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep that is what I thought of, to also pull that in the modules config. I am glad I wasn't completely off here. I'll give it a try probably tomorrow afternoon

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

plus by then maybe the yet new modules syntax is there and I can catch two birds with one stone 😆

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@FriederikeHanssen will you be adding the split_fastq logic in the bwamem modules in #462 or will you be doing that in a separate PR?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll see that I can add it in :)

FriederikeHanssen
FriederikeHanssen approved these changes Dec 2, 2021
View reviewed changes
Copy link
Contributor

@FriederikeHanssen FriederikeHanssen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From my side this looks all great. Taking the 'parameter logic' out of the subworkflow is pretty nice, makes the workflow look a lot simpler and less crowded. I can also now better see how we can make it work with UMI based subworkflow + mapping.

@maxulysse maxulysse merged commit 6f05f41 into nf-core:dev Dec 2, 2021
@maxulysse maxulysse deleted the dev_get_versions branch December 2, 2021 08:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mahesh-panchal mahesh-panchal mahesh-panchal left review comments

@FriederikeHanssen FriederikeHanssen FriederikeHanssen approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@maxulysse @FriederikeHanssen @mahesh-panchal