Clean -fw from sample names in MultiQC #375

ewels · 2020-02-04T09:28:32Z

The new gzipping / rRNA code with --removeRiboRNA in the pipeline now appends -fw and -rv to the FastQ file names:

rnaseq/main.nf

Lines 987 to 988 in 3b6df9b

    
                       gzip < non-rRNA-reads-fw.fq > ${name}-fw.fq.gz 
        
                       gzip < non-rRNA-reads-rv.fq > ${name}-rv.fq.gz

These are not currently trimmed off by MultiQC and result in the sample rows being split:

This PR adds -fw to the MultiQC config, so that the sample names are cleaned and will merge into a single row again.

PR checklist

PR is to dev rather than master
This comment contains a description of changes (with reason)
If you've fixed a bug or added code that should be tested, add tests!
If necessary, also make a PR on the nf-core/rnaseq branch on the nf-core/test-datasets repo
Ensure the test suite passes (nextflow run . -profile test,docker).
Make sure your code lints (nf-core lint .).
Documentation in docs is updated
CHANGELOG.md is updated
README.md is updated

maxulysse

Looks good

kviljoen · 2020-02-05T13:46:48Z

@ewels as discussed on gitter this was for nfcore-rnaseq v 1.4.2 (multiqc v1.7). I have made some changes for implementation on our cluster but mainly to resource settings so should affect the multiqc part (https://github.com/kviljoen/RNAseq)

ewels · 2020-02-07T14:33:54Z

Hi @kviljoen,

You should not need to fork and edit the pipeline code to change configuration settings. The pipelines (and Nextflow) are designed so that you can overwrite just about everything in your own files. Then the codebase is kept "clean" and the pipeline is more reproducible, plus you can easily update without overwriting your changes.

See https://nf-co.re/usage/configuration for some documentation (recently revamped) and https://nf-co.re/usage/configuration as the place to go to add your config profile (which will then be instantly available to all nf-core pipelines).

Phil

Fixes new lint tests to standardise all command line parameters to use snake_case instead of camelCase

ewels · 2020-02-07T17:11:49Z

I'm not sure why the GitHub actions lint test is failing here.. @ggabernet - any ideas?

ggabernet · 2020-02-07T19:22:12Z

.github/workflows/branch.yml

      - name: Check PRs
        run: |
-          [[ $(git remote get-url origin) == *nf-core/rnaseq ]] && [[ ${GITHUB_BASE_REF} = "master" ]] && { [[ ${GITHUB_HEAD_REF} = "dev" ]] || [[ ${GITHUB_BASE_REF} = "patch" ]]; }
+          { [[ $(git remote get-url origin) == *{{cookiecutter.name}} ]] && [[ ${GITHUB_HEAD_REF} = "dev" ]]; } || [[ ${GITHUB_HEAD_REF} == "patch" ]]


Suggested change

{ [[ $(git remote get-url origin) == *{{cookiecutter.name}} ]] && [[ ${GITHUB_HEAD_REF} = "dev" ]]; } || [[ ${GITHUB_HEAD_REF} == "patch" ]]

{ [[ $(git remote get-url origin) == *nf-core/rnaseq ]] && [[ ${GITHUB_HEAD_REF} = "dev" ]]; } || [[ ${GITHUB_HEAD_REF} == "patch" ]]

kviljoen · 2020-06-17T07:52:05Z

Hi @ewels I'm still getting sample rows being split by '-fw' in the multiqc report even after inclusion of '-fw' in the multiqc_config.yaml for stripping.

ewels · 2020-06-17T07:55:05Z

Hi @kviljoen - sorry to hear that. This is definitely with the dev version of the pipeline? I don't think that this change is in a stable release yet..

kviljoen · 2020-06-17T08:09:30Z

Yes @ewels I tried to use the dev version but got an unrelated error: 'Unknown variable 'ch_fasta_for_rsem_reference' where it looks like there was no channel created for ch_fasta_for_rsem_reference? I've therefore just added '-fw' to my own multiqc_config.yaml, not sure if I need any additional edits? Thanks for the quick reply! Katie.

ewels · 2020-06-17T08:48:38Z

Ok, we should make a new issue about the ch_fasta_for_rsem_reference problem 😅

The change you mentioned should be all that is required. You can see that block of the MultiQC config in the pipeline here:

rnaseq/assets/multiqc_config.yaml

Lines 1 to 7 in 8845b21

    
           extra_fn_clean_exts: 
        
               - '_R1' 
        
               - '_R2' 
        
               - '.hisat' 
        
               - '_subsamp' 
        
               - '.sorted' 
        
               - '-fw'

It seems to work fine in my testing though 🤔

kviljoen · 2020-06-17T09:35:05Z

Thanks @ewels probably something on my side then. I'm currently using process.container = 'nfcore/rnaseq:1.4.2' not sure if that would change anything. Not sure if the ch_fasta_for_rsem_reference problem is on my side, perhaps you could check if working for you? Regards, Katie.

ewels · 2020-06-17T10:05:21Z

If you're using the development code, you should be using the development container - the dev tag instead of 1.4.2. This tag should be hardcoded in the pipeline though, usually there's not a need to specify this yourself?

The automated tests are passing fine so it must be something specific to the way that you're running the pipeline. Let's discuss on the other issue thread though.

kviljoen · 2020-06-17T10:30:08Z

Hi Phil, yes, dev container specified for dev branch (I was referring to my master branch where I'm on 1.4.2 and had only changed -fw in multiqc_config.yaml). Attached log file from dev run. Please note that this is a result of merging with my own version of the pipeline because we've had to make some small tweaks particularly with java-related process memory settings in the main.nf to get it to run optimally on our cluster so may be specific to my version of pipeline. Log file attached
testrun.log

ewels · 2020-06-17T10:39:19Z

Hi @kviljoen,

I can't see any MultiQC config edits when comparing your fork?

Will move discussion of the log over to the other issue to try to keep these two separate 😅

Phil

kviljoen · 2020-06-17T10:57:08Z

Hi Phil,

The test I ran was on my master branch not dev, sorry (since dev wasn't working because of the other ch_fasta_for_rsem_reference issue). It's here

Regards,
Katie

ewels · 2020-06-17T11:10:38Z

Ok, the change looks fine. And you're not supplying a custom MultiQC config or anything? Is it possible to attach a MultiQC report where this is a problem?

kviljoen · 2020-06-17T11:26:59Z

Hi Phil, no custom MultiQC config no.

kviljoen · 2020-07-28T10:31:13Z

Hi @ewels I'm now running the nf-core/dev pipeline but am still having this issue, unfortunately.

Clean -fw from sample names in MultiQC

9b18819

ewels requested a review from a team February 4, 2020 09:28

ewels added the enhancement label Feb 4, 2020

ewels mentioned this pull request Feb 4, 2020

Clean -fw from sample names in MultiQC #374

Closed

9 tasks

maxulysse approved these changes Feb 4, 2020

View reviewed changes

ewels added 2 commits February 7, 2020 15:41

Update singleEnd and igenomesIgnore to snake_case

6b9d769

Fixes new lint tests to standardise all command line parameters to use snake_case instead of camelCase

Update CI branch actions file to match the template

a927f89

ggabernet reviewed Feb 7, 2020

View reviewed changes

apeltzer changed the base branch from dev to merge-stuff February 11, 2020 19:08

apeltzer merged commit b4748ad into nf-core:merge-stuff Feb 11, 2020

ewels deleted the multiqc-clean-fw branch June 17, 2020 07:55

ewels mentioned this pull request Jun 17, 2020

No channel created for ch_fasta_for_rsem_reference #432

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Clean -fw from sample names in MultiQC #375

Clean -fw from sample names in MultiQC #375

ewels commented Feb 4, 2020

maxulysse left a comment

kviljoen commented Feb 5, 2020

ewels commented Feb 7, 2020

ewels commented Feb 7, 2020

ggabernet Feb 7, 2020

ewels Feb 8, 2020

kviljoen commented Jun 17, 2020

ewels commented Jun 17, 2020

kviljoen commented Jun 17, 2020 via email •

edited by ewels

Loading

ewels commented Jun 17, 2020

kviljoen commented Jun 17, 2020 via email •

edited by ewels

Loading

ewels commented Jun 17, 2020

kviljoen commented Jun 17, 2020 •

edited

Loading

ewels commented Jun 17, 2020

kviljoen commented Jun 17, 2020

ewels commented Jun 17, 2020

kviljoen commented Jun 17, 2020

kviljoen commented Jul 28, 2020

	gzip < non-rRNA-reads-fw.fq > ${name}-fw.fq.gz
	gzip < non-rRNA-reads-rv.fq > ${name}-rv.fq.gz

	{ [[ $(git remote get-url origin) == *{{cookiecutter.name}} ]] && [[ ${GITHUB_HEAD_REF} = "dev" ]]; } \|\| [[ ${GITHUB_HEAD_REF} == "patch" ]]
	{ [[ $(git remote get-url origin) == *nf-core/rnaseq ]] && [[ ${GITHUB_HEAD_REF} = "dev" ]]; } \|\| [[ ${GITHUB_HEAD_REF} == "patch" ]]

Clean -fw from sample names in MultiQC #375

Clean -fw from sample names in MultiQC #375

Conversation

ewels commented Feb 4, 2020

PR checklist

maxulysse left a comment

Choose a reason for hiding this comment

kviljoen commented Feb 5, 2020

ewels commented Feb 7, 2020

ewels commented Feb 7, 2020

ggabernet Feb 7, 2020

Choose a reason for hiding this comment

ewels Feb 8, 2020

Choose a reason for hiding this comment

kviljoen commented Jun 17, 2020

ewels commented Jun 17, 2020

kviljoen commented Jun 17, 2020 via email • edited by ewels Loading

ewels commented Jun 17, 2020

kviljoen commented Jun 17, 2020 via email • edited by ewels Loading

ewels commented Jun 17, 2020

kviljoen commented Jun 17, 2020 • edited Loading

ewels commented Jun 17, 2020

kviljoen commented Jun 17, 2020

ewels commented Jun 17, 2020

kviljoen commented Jun 17, 2020

kviljoen commented Jul 28, 2020

kviljoen commented Jun 17, 2020 via email •

edited by ewels

Loading

kviljoen commented Jun 17, 2020 via email •

edited by ewels

Loading

kviljoen commented Jun 17, 2020 •

edited

Loading