Merge pull request #85 from nf-core/dev

PR for release 1.1.0

.github/.dockstore.yml

@@ -0,0 +1,6 @@
+# Dockstore config version, not pipeline version
+version: 1.2
+workflows:
+  - subclass: nfl
+    primaryDescriptorPath: /nextflow.config
+    publish: True

.github/CONTRIBUTING.md

@@ -1,47 +1,128 @@
 # nf-core/smrnaseq: Contributing Guidelines
 
-Hi there! Many thanks for taking an interest in improving nf-core/smrnaseq.
+Hi there!
+Many thanks for taking an interest in improving nf-core/smrnaseq.
 
-We try to manage the required tasks for nf-core/smrnaseq using GitHub issues, you probably came to this page when creating one. Please use the pre-filled template to save time.
-
-However, don't be put off by this template - other more general issues and suggestions are welcome! Contributions to the code are even more welcome ;)
-
-> If you need help using or modifying nf-core/smrnaseq then the best place to ask is on the pipeline channel on [Slack](https://nf-core-invite.herokuapp.com/).
+We try to manage the required tasks for nf-core/smrnaseq using GitHub issues, you probably came to this page when creating one.
+Please use the pre-filled template to save time.
 
+However, don't be put off by this template - other more general issues and suggestions are welcome!
+Contributions to the code are even more welcome ;)
 
+> If you need help using or modifying nf-core/smrnaseq then the best place to ask is on the nf-core Slack [#smrnaseq](https://nfcore.slack.com/channels/smrnaseq) channel ([join our Slack here](https://nf-co.re/join/slack)).
 
 ## Contribution workflow
-If you'd like to write some code for nf-core/smrnaseq, the standard workflow
-is as follows:
 
-1. Check that there isn't already an issue about your idea in the
-   [nf-core/smrnaseq issues](https://github.com/nf-core/smrnaseq/issues) to avoid
-   duplicating work.
-    * If there isn't one already, please create one so that others know you're working on this
-2. Fork the [nf-core/smrnaseq repository](https://github.com/nf-core/smrnaseq) to your GitHub account
-3. Make the necessary changes / additions within your forked repository
-4. Submit a Pull Request against the `dev` branch and wait for the code to be reviewed and merged.
+If you'd like to write some code for nf-core/smrnaseq, the standard workflow is as follows:
 
-If you're not used to this workflow with git, you can start with some [basic docs from GitHub](https://help.github.com/articles/fork-a-repo/) or even their [excellent interactive tutorial](https://try.github.io/).
+1. Check that there isn't already an issue about your idea in the [nf-core/smrnaseq issues](https://github.com/nf-core/smrnaseq/issues) to avoid duplicating work
+    * If there isn't one already, please create one so that others know you're working on this
+2. [Fork](https://help.github.com/en/github/getting-started-with-github/fork-a-repo) the [nf-core/smrnaseq repository](https://github.com/nf-core/smrnaseq) to your GitHub account
+3. Make the necessary changes / additions within your forked repository following [Pipeline conventions](#pipeline-contribution-conventions)
+4. Use `nf-core schema build .` and add any new parameters to the pipeline JSON schema (requires [nf-core tools](https://github.com/nf-core/tools) >= 1.10).
+5. Submit a Pull Request against the `dev` branch and wait for the code to be reviewed and merged
 
+If you're not used to this workflow with git, you can start with some [docs from GitHub](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests) or even their [excellent `git` resources](https://try.github.io/).
 
 ## Tests
-When you create a pull request with changes, [Travis CI](https://travis-ci.org/) will run automatic tests.
+
+When you create a pull request with changes, [GitHub Actions](https://github.com/features/actions) will run automatic tests.
 Typically, pull-requests are only fully reviewed when these tests are passing, though of course we can help out before then.
 
 There are typically two types of tests that run:
 
-### Lint Tests
-The nf-core has a [set of guidelines](http://nf-co.re/guidelines) which all pipelines must adhere to.
+### Lint tests
+
+`nf-core` has a [set of guidelines](https://nf-co.re/developers/guidelines) which all pipelines must adhere to.
 To enforce these and ensure that all pipelines stay in sync, we have developed a helper tool which runs checks on the pipeline code. This is in the [nf-core/tools repository](https://github.com/nf-core/tools) and once installed can be run locally with the `nf-core lint <pipeline-directory>` command.
 
 If any failures or warnings are encountered, please follow the listed URL for more documentation.
 
-### Pipeline Tests
-Each nf-core pipeline should be set up with a minimal set of test-data.
-Travis CI then runs the pipeline on this data to ensure that it exists successfully.
+### Pipeline tests
+
+Each `nf-core` pipeline should be set up with a minimal set of test-data.
+`GitHub Actions` then runs the pipeline on this data to ensure that it exits successfully.
 If there are any failures then the automated tests fail.
-These tests are run both with the latest available version of Nextflow and also the minimum required version that is stated in the pipeline code.
+These tests are run both with the latest available version of `Nextflow` and also the minimum required version that is stated in the pipeline code.
+
+## Patch
+
+:warning: Only in the unlikely and regretful event of a release happening with a bug.
+
+* On your own fork, make a new branch `patch` based on `upstream/master`.
+* Fix the bug, and bump version (X.Y.Z+1).
+* A PR should be made on `master` from patch to directly this particular bug.
 
 ## Getting help
-For further information/help, please consult the [nf-core/smrnaseq documentation](https://github.com/nf-core/smrnaseq#documentation) and don't hesitate to get in touch on the pipeline channel on [Slack](https://nf-core-invite.herokuapp.com/).
+
+For further information/help, please consult the [nf-core/smrnaseq documentation](https://nf-co.re/smrnaseq/usage) and don't hesitate to get in touch on the nf-core Slack [#smrnaseq](https://nfcore.slack.com/channels/smrnaseq) channel ([join our Slack here](https://nf-co.re/join/slack)).
+
+## Pipeline contribution conventions
+
+To make the nf-core/smrnaseq code and processing logic more understandable for new contributors and to ensure quality, we semi-standardise the way the code and other contributions are written.
+
+### Adding a new step
+
+If you wish to contribute a new step, please use the following coding standards:
+
+1. Define the corresponding input channel into your new process from the expected previous process channel
+2. Write the process block (see below).
+3. Define the output channel if needed (see below).
+4. Add any new flags/options to `nextflow.config` with a default (see below).
+5. Add any new flags/options to `nextflow_schema.json` with help text (with `nf-core schema build .`).
+6. Add any new flags/options to the help message (for integer/text parameters, print to help the corresponding `nextflow.config` parameter).
+7. Add sanity checks for all relevant parameters.
+8. Add any new software to the `scrape_software_versions.py` script in `bin/` and the version command to the `scrape_software_versions` process in `main.nf`.
+9. Do local tests that the new code works properly and as expected.
+10. Add a new test command in `.github/workflow/ci.yaml`.
+11. If applicable add a [MultiQC](https://https://multiqc.info/) module.
+12. Update MultiQC config `assets/multiqc_config.yaml` so relevant suffixes, name clean up, General Statistics Table column order, and module figures are in the right order.
+13. Optional: Add any descriptions of MultiQC report sections and output files to `docs/output.md`.
+
+### Default values
+
+Parameters should be initialised / defined with default values in `nextflow.config` under the `params` scope.
+
+Once there, use `nf-core schema build .` to add to `nextflow_schema.json`.
+
+### Default processes resource requirements
+
+Sensible defaults for process resource requirements (CPUs / memory / time) for a process should be defined in `conf/base.config`. These should generally be specified generic with `withLabel:` selectors so they can be shared across multiple processes/steps of the pipeline. A nf-core standard set of labels that should be followed where possible can be seen in the [nf-core pipeline template](https://github.com/nf-core/tools/blob/master/nf_core/pipeline-template/conf/base.config), which has the default process as a single core-process, and then different levels of multi-core configurations for increasingly large memory requirements defined with standardised labels.
+
+The process resources can be passed on to the tool dynamically within the process with the `${task.cpu}` and `${task.memory}` variables in the `script:` block.
+
+### Naming schemes
+
+Please use the following naming schemes, to make it easy to understand what is going where.
+
+* initial process channel: `ch_output_from_<process>`
+* intermediate and terminal channels: `ch_<previousprocess>_for_<nextprocess>`
+
+### Nextflow version bumping
+
+If you are using a new feature from core Nextflow, you may bump the minimum required version of nextflow in the pipeline with: `nf-core bump-version --nextflow . [min-nf-version]`
+
+### Software version reporting
+
+If you add a new tool to the pipeline, please ensure you add the information of the tool to the `get_software_version` process.
+
+Add to the script block of the process, something like the following:
+
+```bash
+<YOUR_TOOL> --version &> v_<YOUR_TOOL>.txt 2>&1 || true
+```
+
+or
+
+```bash
+<YOUR_TOOL> --help | head -n 1 &> v_<YOUR_TOOL>.txt 2>&1 || true
+```
+
+You then need to edit the script `bin/scrape_software_versions.py` to:
+
+1. Add a Python regex for your tool's `--version` output (as in stored in the `v_<YOUR_TOOL>.txt` file), to ensure the version is reported as a `v` and the version number e.g. `v2.1.1`
+2. Add a HTML entry to the `OrderedDict` for formatting in MultiQC.
+
+### Images and figures
+
+For overview images and other documents we follow the nf-core [style guidelines and examples](https://nf-co.re/developers/design_guidelines).

.github/ISSUE_TEMPLATE/bug_report.md

@@ -1,31 +1,64 @@
+---
+name: Bug report
+about: Report something that is broken or incorrect
+labels: bug
+---
+
+<!--
+# nf-core/smrnaseq bug report
+
 Hi there!
 
-Thanks for telling us about a problem with the pipeline. Please delete this text and anything that's not relevant from the template below:
+Thanks for telling us about a problem with the pipeline.
+Please delete this text and anything that's not relevant from the template below:
+-->
+
+## Check Documentation
+
+I have checked the following places for your error:
 
-#### Describe the bug
-A clear and concise description of what the bug is.
+- [ ] [nf-core website: troubleshooting](https://nf-co.re/usage/troubleshooting)
+- [ ] [nf-core/smrnaseq pipeline documentation](https://nf-co.re/nf-core/smrnaseq/usage)
+
+## Description of the bug
+
+<!-- A clear and concise description of what the bug is. -->
+
+## Steps to reproduce
 
-#### Steps to reproduce
 Steps to reproduce the behaviour:
-1. Command line: `nextflow run ...`
-2. See error: _Please provide your error message_
 
-#### Expected behaviour
-A clear and concise description of what you expected to happen.
+1. Command line: <!-- [e.g. `nextflow run ...`] -->
+2. See error: <!-- [Please provide your error message] -->
+
+## Expected behaviour
+
+<!-- A clear and concise description of what you expected to happen. -->
+
+## Log files
+
+Have you provided the following extra information/files:
+
+- [ ] The command used to run the pipeline
+- [ ] The `.nextflow.log` file <!-- this is a hidden file in the directory where you launched the pipeline -->
+
+## System
+
+- Hardware: <!-- [e.g. HPC, Desktop, Cloud...] -->
+- Executor: <!-- [e.g. slurm, local, awsbatch...] -->
+- OS: <!-- [e.g. CentOS Linux, macOS, Linux Mint...] -->
+- Version <!-- [e.g. 7, 10.13.6, 18.3...] -->
+
+## Nextflow Installation
+
+- Version: <!-- [e.g. 19.10.0] -->
 
-#### System:
- - Hardware: [e.g. HPC, Desktop, Cloud...]
- - Executor: [e.g. slurm, local, awsbatch...]
- - OS: [e.g. CentOS Linux, macOS, Linux Mint...]
- - Version [e.g. 7, 10.13.6, 18.3...]
+## Container engine
 
-#### Nextflow Installation:
- - Version: [e.g. 0.31.0]
+- Engine: <!-- [e.g. Conda, Docker, Singularity, Podman, Shifter or Charliecloud] -->
+- version: <!-- [e.g. 1.0.0] -->
+- Image tag: <!-- [e.g. nfcore/smrnaseq:1.0.0] -->
 
-#### Container engine:
- - Engine: [e.g. Conda, Docker or Singularity]
- - version: [e.g. 1.0.0]
- - Image tag: [e.g. nfcore/smrnaseq:1.0.0]
+## Additional context
 
-#### Additional context
-Add any other context about the problem here.
+<!-- Add any other context about the problem here. -->

.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1,8 @@
+blank_issues_enabled: false
+contact_links:
+  - name: Join nf-core
+    url: https://nf-co.re/join
+    about: Please join the nf-core community here
+  - name: "Slack #smrnaseq channel"
+    url: https://nfcore.slack.com/channels/smrnaseq
+    about: Discussion about the nf-core/smrnaseq pipeline

.github/ISSUE_TEMPLATE/feature_request.md

@@ -1,16 +1,32 @@
+---
+name: Feature request
+about: Suggest an idea for the nf-core/smrnaseq pipeline
+labels: enhancement
+---
+
+<!--
+# nf-core/smrnaseq feature request
+
 Hi there!
 
-Thanks for suggesting a new feature for the pipeline! Please delete this text and anything that's not relevant from the template below:
+Thanks for suggesting a new feature for the pipeline!
+Please delete this text and anything that's not relevant from the template below:
+-->
+
+## Is your feature request related to a problem? Please describe
+
+<!-- A clear and concise description of what the problem is. -->
+
+<!-- e.g. [I'm always frustrated when ...] -->
+
+## Describe the solution you'd like
+
+<!-- A clear and concise description of what you want to happen. -->
 
-#### Is your feature request related to a problem? Please describe.
-A clear and concise description of what the problem is.
-Ex. I'm always frustrated when [...]
+## Describe alternatives you've considered
 
-#### Describe the solution you'd like
-A clear and concise description of what you want to happen.
+<!-- A clear and concise description of any alternative solutions or features you've considered. -->
 
-#### Describe alternatives you've considered
-A clear and concise description of any alternative solutions or features you've considered.
+## Additional context
 
-#### Additional context
-Add any other context about the feature request here.
+<!-- Add any other context about the feature request here. -->

.github/PULL_REQUEST_TEMPLATE.md

@@ -1,15 +1,27 @@
-Many thanks to contributing to nf-core/smrnaseq!
+<!--
+# nf-core/smrnaseq pull request
 
-Please fill in the appropriate checklist below (delete whatever is not relevant). These are the most common things requested on pull requests (PRs).
+Many thanks for contributing to nf-core/smrnaseq!
+
+Please fill in the appropriate checklist below (delete whatever is not relevant).
+These are the most common things requested on pull requests (PRs).
+
+Remember that PRs should be made against the dev branch, unless you're preparing a pipeline release.
+
+Learn more about contributing: [CONTRIBUTING.md](https://github.com/nf-core/smrnaseq/tree/master/.github/CONTRIBUTING.md)
+-->
+<!-- markdownlint-disable ul-indent -->
 
 ## PR checklist
- - [ ] This comment contains a description of changes (with reason)
- - [ ] If you've fixed a bug or added code that should be tested, add tests!
- - [ ] If necessary, also make a PR on the [nf-core/smrnaseq branch on the nf-core/test-datasets repo]( https://github.com/nf-core/test-datasets/pull/new/nf-core/smrnaseq)
- - [ ] Ensure the test suite passes (`nextflow run . -profile test,docker`).
- - [ ] Make sure your code lints (`nf-core lint .`).
- - [ ] Documentation in `docs` is updated
- - [ ] `CHANGELOG.md` is updated
- - [ ] `README.md` is updated
-
-**Learn more about contributing:** https://github.com/nf-core/smrnaseq/tree/master/.github/CONTRIBUTING.md
+
+- [ ] This comment contains a description of changes (with reason).
+- [ ] If you've fixed a bug or added code that should be tested, add tests!
+    - [ ] If you've added a new tool - add to the software_versions process and a regex to `scrape_software_versions.py`
+    - [ ] If you've added a new tool - have you followed the pipeline conventions in the [contribution docs](<https://github.com/>nf-core/smrnaseq/tree/master/.github/CONTRIBUTING.md)
+    - [ ] If necessary, also make a PR on the nf-core/smrnaseq _branch_ on the [nf-core/test-datasets](https://github.com/nf-core/test-datasets) repository.
+- [ ] Make sure your code lints (`nf-core lint .`).
+- [ ] Ensure the test suite passes (`nextflow run . -profile test,docker`).
+- [ ] Usage Documentation in `docs/usage.md` is updated.
+- [ ] Output Documentation in `docs/output.md` is updated.
+- [ ] `CHANGELOG.md` is updated.
+- [ ] `README.md` is updated (including new tool citations and authors/contributors).

.github/markdownlint.yml

@@ -1,9 +1,12 @@
 # Markdownlint configuration file
-default: true,
+default: true
 line-length: false
-no-multiple-blanks: 0
-blanks-around-headers: false
-blanks-around-lists: false
-header-increment: false
 no-duplicate-header:
     siblings_only: true
+no-inline-html:
+    allowed_elements:
+        - img
+        - p
+        - kbd
+        - details
+        - summary

.github/workflows/awsfulltest.yml

@@ -0,0 +1,43 @@
+name: nf-core AWS full size tests
+# This workflow is triggered on published releases.
+# It can be additionally triggered manually with GitHub actions workflow dispatch.
+# It runs the -profile 'test_full' on AWS batch
+
+on:
+  workflow_run:
+    workflows: ["nf-core Docker push (release)"]
+    types: [completed]
+  workflow_dispatch:
+
+env:
+  AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
+  AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+  TOWER_ACCESS_TOKEN: ${{ secrets.AWS_TOWER_TOKEN }}
+  AWS_JOB_DEFINITION: ${{ secrets.AWS_JOB_DEFINITION }}
+  AWS_JOB_QUEUE: ${{ secrets.AWS_JOB_QUEUE }}
+  AWS_S3_BUCKET: ${{ secrets.AWS_S3_BUCKET }}
+
+jobs:
+  run-awstest:
+    name: Run AWS full tests
+    if: github.repository == 'nf-core/smrnaseq'
+    runs-on: ubuntu-latest
+    steps:
+      - name: Setup Miniconda
+        uses: conda-incubator/setup-miniconda@v2
+        with:
+          auto-update-conda: true
+          python-version: 3.7
+      - name: Install awscli
+        run: conda install -c conda-forge awscli
+      - name: Start AWS batch job
+        # Add full size test data (but still relatively small datasets for few samples)
+        # on the `test_full.config` test runs with only one set of parameters
+        # Then specify `-profile test_full` instead of `-profile test` on the AWS batch command
+        run: |
+          aws batch submit-job \
+            --region eu-west-1 \
+            --job-name nf-core-smrnaseq \
+            --job-queue $AWS_JOB_QUEUE \
+            --job-definition $AWS_JOB_DEFINITION \
+            --container-overrides '{"command": ["nf-core/smrnaseq", "-r '"${GITHUB_SHA}"' -profile test_full --outdir s3://'"${AWS_S3_BUCKET}"'/smrnaseq/results-'"${GITHUB_SHA}"' -w s3://'"${AWS_S3_BUCKET}"'/smrnaseq/work-'"${GITHUB_SHA}"' -with-tower"], "environment": [{"name": "TOWER_ACCESS_TOKEN", "value": "'"$TOWER_ACCESS_TOKEN"'"}]}'