Merge pull request #12 from qbic-pipelines/dev

Release 1.0

.github/.dockstore.yml

@@ -3,3 +3,4 @@ version: 1.2
 workflows:
   - subclass: nfl
     primaryDescriptorPath: /nextflow.config
+    publish: True

.github/CONTRIBUTING.md

@@ -28,14 +28,14 @@ Typically, pull-requests are only fully reviewed when these tests are passing, t
 
 There are typically two types of tests that run:
 
-### Lint Tests
+### Lint tests
 
 `nf-core` has a [set of guidelines](https://nf-co.re/developers/guidelines) which all pipelines must adhere to.
 To enforce these and ensure that all pipelines stay in sync, we have developed a helper tool which runs checks on the pipeline code. This is in the [nf-core/tools repository](https://github.com/nf-core/tools) and once installed can be run locally with the `nf-core lint <pipeline-directory>` command.
 
 If any failures or warnings are encountered, please follow the listed URL for more documentation.
 
-### Pipeline Tests
+### Pipeline tests
 
 Each `nf-core` pipeline should be set up with a minimal set of test-data.
 `GitHub Actions` then runs the pipeline on this data to ensure that it exits successfully.
@@ -52,4 +52,74 @@ These tests are run both with the latest available version of `Nextflow` and als
 
 ## Getting help
 
-For further information/help, please consult the [qbic-pipelines/cellranger documentation](./docs/usage.md) and don't hesitate to get in touch.
+For further information/help, please consult the [qbic-pipelines/cellranger documentation](./docs/usage.md).
+
+## Pipeline contribution conventions
+
+To make the qbic-pipelines/cellranger code and processing logic more understandable for new contributors and to ensure quality, we semi-standardise the way the code and other contributions are written.
+
+### Adding a new step
+
+If you wish to contribute a new step, please use the following coding standards:
+
+1. Define the corresponding input channel into your new process from the expected previous process channel
+2. Write the process block (see below).
+3. Define the output channel if needed (see below).
+4. Add any new flags/options to `nextflow.config` with a default (see below).
+5. Add any new flags/options to `nextflow_schema.json` with help text (with `nf-core schema build .`).
+6. Add any new flags/options to the help message (for integer/text parameters, print to help the corresponding `nextflow.config` parameter).
+7. Add sanity checks for all relevant parameters.
+8. Add any new software to the `scrape_software_versions.py` script in `bin/` and the version command to the `scrape_software_versions` process in `main.nf`.
+9. Do local tests that the new code works properly and as expected.
+10. Add a new test command in `.github/workflow/ci.yaml`.
+11. If applicable add a [MultiQC](https://https://multiqc.info/) module.
+12. Update MultiQC config `assets/multiqc_config.yaml` so relevant suffixes, name clean up, General Statistics Table column order, and module figures are in the right order.
+13. Optional: Add any descriptions of MultiQC report sections and output files to `docs/output.md`.
+
+### Default values
+
+Parameters should be initialised / defined with default values in `nextflow.config` under the `params` scope.
+
+Once there, use `nf-core schema build .` to add to `nextflow_schema.json`.
+
+### Default processes resource requirements
+
+Sensible defaults for process resource requirements (CPUs / memory / time) for a process should be defined in `conf/base.config`. These should generally be specified generic with `withLabel:` selectors so they can be shared across multiple processes/steps of the pipeline. A nf-core standard set of labels that should be followed where possible can be seen in the [nf-core pipeline template](https://github.com/nf-core/tools/blob/master/nf_core/pipeline-template/conf/base.config), which has the default process as a single core-process, and then different levels of multi-core configurations for increasingly large memory requirements defined with standardised labels.
+
+The process resources can be passed on to the tool dynamically within the process with the `${task.cpu}` and `${task.memory}` variables in the `script:` block.
+
+### Naming schemes
+
+Please use the following naming schemes, to make it easy to understand what is going where.
+
+* initial process channel: `ch_output_from_<process>`
+* intermediate and terminal channels: `ch_<previousprocess>_for_<nextprocess>`
+
+### Nextflow version bumping
+
+If you are using a new feature from core Nextflow, you may bump the minimum required version of nextflow in the pipeline with: `nf-core bump-version --nextflow . [min-nf-version]`
+
+### Software version reporting
+
+If you add a new tool to the pipeline, please ensure you add the information of the tool to the `get_software_version` process.
+
+Add to the script block of the process, something like the following:
+
+```bash
+<YOUR_TOOL> --version &> v_<YOUR_TOOL>.txt 2>&1 || true
+```
+
+or
+
+```bash
+<YOUR_TOOL> --help | head -n 1 &> v_<YOUR_TOOL>.txt 2>&1 || true
+```
+
+You then need to edit the script `bin/scrape_software_versions.py` to:
+
+1. Add a Python regex for your tool's `--version` output (as in stored in the `v_<YOUR_TOOL>.txt` file), to ensure the version is reported as a `v` and the version number e.g. `v2.1.1`
+2. Add a HTML entry to the `OrderedDict` for formatting in MultiQC.
+
+### Images and figures
+
+For overview images and other documents we follow the nf-core [style guidelines and examples](https://nf-co.re/developers/design_guidelines).

.github/ISSUE_TEMPLATE/bug_report.md

@@ -13,6 +13,13 @@ Thanks for telling us about a problem with the pipeline.
 Please delete this text and anything that's not relevant from the template below:
 -->
 
+## Check Documentation
+
+I have checked the following places for your error:
+
+- [ ] [nf-core website: troubleshooting](https://nf-co.re/usage/troubleshooting)
+- [ ] [qbic-pipelines/cellranger pipeline documentation](https://github.com/qbic-pipelines/cellranger/blob/master/docs/usage.md)
+
 ## Description of the bug
 
 <!-- A clear and concise description of what the bug is. -->
@@ -28,6 +35,13 @@ Steps to reproduce the behaviour:
 
 <!-- A clear and concise description of what you expected to happen. -->
 
+## Log files
+
+Have you provided the following extra information/files:
+
+- [ ] The command used to run the pipeline
+- [ ] The `.nextflow.log` file <!-- this is a hidden file in the directory where you launched the pipeline -->
+
 ## System
 
 - Hardware: <!-- [e.g. HPC, Desktop, Cloud...] -->
@@ -41,9 +55,9 @@ Steps to reproduce the behaviour:
 
 ## Container engine
 
-- Engine: <!-- [e.g. Conda, Docker, Singularity or Podman] -->
+- Engine: <!-- [e.g. Conda, Docker, Singularity, Podman, Shifter or Charliecloud] -->
 - version: <!-- [e.g. 1.0.0] -->
-- Image tag: <!-- [e.g. nfcore/cellranger:1.0.0] -->
+- Image tag: <!-- [e.g. qbicpipelines/cellranger:1.0.0] -->
 
 ## Additional context
 

.github/PULL_REQUEST_TEMPLATE.md

@@ -10,10 +10,18 @@ Remember that PRs should be made against the dev branch, unless you're preparing
 
 Learn more about contributing: [CONTRIBUTING.md](https://github.com/qbic-pipelines/cellranger/tree/master/.github/CONTRIBUTING.md)
 -->
+<!-- markdownlint-disable ul-indent -->
 
 ## PR checklist
 
-- [ ] This comment contains a description of changes (with reason)
-- [ ] `CHANGELOG.md` is updated
+- [ ] This comment contains a description of changes (with reason).
 - [ ] If you've fixed a bug or added code that should be tested, add tests!
-- [ ] Documentation in `docs` is updated
+    - [ ] If you've added a new tool - add to the software_versions process and a regex to `scrape_software_versions.py`
+    - [ ] If you've added a new tool - have you followed the pipeline conventions in the [contribution docs](<https://github.com/qbic-pipelines/cellranger/tree/master/.github/CONTRIBUTING.md)
+    - [ ] If necessary, also make a PR on the nf-core/qbic-pipelines-cellranger _branch_ on the [nf-core/test-datasets](https://github.com/nf-core/test-datasets) repository.
+- [ ] Make sure your code lints (`nf-core lint .`).
+- [ ] Ensure the test suite passes (`nextflow run . -profile test,docker`).
+- [ ] Usage Documentation in `docs/usage.md` is updated.
+- [ ] Output Documentation in `docs/output.md` is updated.
+- [ ] `CHANGELOG.md` is updated.
+- [ ] `README.md` is updated (including new tool citations and authors/contributors).

.github/markdownlint.yml

@@ -1,5 +1,12 @@
 # Markdownlint configuration file
-default: true,
+default: true
 line-length: false
 no-duplicate-header:
     siblings_only: true
+no-inline-html:
+    allowed_elements:
+        - img
+        - p
+        - kbd
+        - details
+        - summary

.github/workflows/branch.yml

@@ -23,13 +23,22 @@ jobs:
         uses: mshick/add-pr-comment@v1
         with:
           message: |
+            ## This PR is against the `master` branch :x:
+
+            * Do not close this PR
+            * Click _Edit_ and change the `base` to `dev`
+            * This CI test will remain failed until you push a new commit
+
+            ---
+
             Hi @${{ github.event.pull_request.user.login }},
 
             It looks like this pull-request is has been made against the ${{github.event.pull_request.head.repo.full_name}} `master` branch.
             The `master` branch on qbic-pipelines repositories should always contain code from the latest release.
             Because of this, PRs to `master` are only allowed if they come from the ${{github.event.pull_request.head.repo.full_name}} `dev` branch.
 
             You do not need to close this PR, you can change the target branch to `dev` by clicking the _"Edit"_ button at the top of this page.
+            Note that even after this, the test will continue to show as failing until you push a new commit.
 
             Thanks again for your contribution!
           repo-token: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/ci.yml

@@ -8,6 +8,9 @@ on:
   release:
     types: [published]
 
+# Uncomment if we need an edge release of Nextflow again
+# env: NXF_EDGE: 1
+
 jobs:
   test:
     name: Run workflow tests
@@ -20,15 +23,15 @@ jobs:
     strategy:
       matrix:
         # Nextflow versions: check pipeline minimum and current latest
-        nxf_ver: ['20.04.0', '']
+        nxf_ver: ['20.10.0', '']
     steps:
       - name: Check out pipeline code
         uses: actions/checkout@v2
 
       - name: Pull docker image
         run: |
           docker pull qbicpipelines/cellranger:dev
-          docker tag qbicpipelines/cellranger:dev qbicpipelines/cellranger:dev
+          docker tag qbicpipelines/cellranger:dev qbicpipelines/cellranger:1.0
 
       - name: Install Nextflow
         env:
@@ -38,8 +41,5 @@ jobs:
           sudo mv nextflow /usr/local/bin/
 
       - name: Run pipeline with test data
-        # TODO nf-core: You can customise CI pipeline run tests as required
-        # For example: adding multiple test runs with different parameters
-        # Remember that you can parallelise this by using strategy.matrix
         run: |
           nextflow run ${GITHUB_WORKSPACE} -profile test,docker

.github/workflows/linting.yml

@@ -19,6 +19,34 @@ jobs:
         run: npm install -g markdownlint-cli
       - name: Run Markdownlint
         run: markdownlint ${GITHUB_WORKSPACE} -c ${GITHUB_WORKSPACE}/.github/markdownlint.yml
+
+      # If the above check failed, post a comment on the PR explaining the failure
+      - name: Post PR comment
+        if: failure()
+        uses: mshick/add-pr-comment@v1
+        with:
+          message: |
+            ## Markdown linting is failing
+
+            To keep the code consistent with lots of contributors, we run automated code consistency checks.
+            To fix this CI test, please run:
+
+            * Install `markdownlint-cli`
+                * On Mac: `brew install markdownlint-cli`
+                * Everything else: [Install `npm`](https://www.npmjs.com/get-npm) then [install `markdownlint-cli`](https://www.npmjs.com/package/markdownlint-cli) (`npm install -g markdownlint-cli`)
+            * Fix the markdown errors
+                * Automatically: `markdownlint . --config .github/markdownlint.yml --fix`
+                * Manually resolve anything left from `markdownlint . --config .github/markdownlint.yml`
+
+            Once you push these changes the test should pass, and you can hide this comment :+1:
+
+            We highly recommend setting up markdownlint in your code editor so that this formatting is done automatically on save. Ask about it on Slack for help!
+
+            Thanks again for your contribution!
+          repo-token: ${{ secrets.GITHUB_TOKEN }}
+          allow-repeats: false
+
+
   YAML:
     runs-on: ubuntu-latest
     steps:
@@ -29,7 +57,34 @@ jobs:
       - name: Install yaml-lint
         run: npm install -g yaml-lint
       - name: Run yaml-lint
-        run: yamllint $(find ${GITHUB_WORKSPACE} -type f -name "*.yml")
+        run: yamllint $(find ${GITHUB_WORKSPACE} -type f -name "*.yml" -o -name "*.yaml")
+
+      # If the above check failed, post a comment on the PR explaining the failure
+      - name: Post PR comment
+        if: failure()
+        uses: mshick/add-pr-comment@v1
+        with:
+          message: |
+            ## YAML linting is failing
+
+            To keep the code consistent with lots of contributors, we run automated code consistency checks.
+            To fix this CI test, please run:
+
+            * Install `yaml-lint`
+                * [Install `npm`](https://www.npmjs.com/get-npm) then [install `yaml-lint`](https://www.npmjs.com/package/yaml-lint) (`npm install -g yaml-lint`)
+            * Fix the markdown errors
+                * Run the test locally: `yamllint $(find . -type f -name "*.yml" -o -name "*.yaml")`
+                * Fix any reported errors in your YAML files
+
+            Once you push these changes the test should pass, and you can hide this comment :+1:
+
+            We highly recommend setting up yaml-lint in your code editor so that this formatting is done automatically on save. Ask about it on Slack for help!
+
+            Thanks again for your contribution!
+          repo-token: ${{ secrets.GITHUB_TOKEN }}
+          allow-repeats: false
+
+
   nf-core:
     runs-on: ubuntu-latest
     steps:
@@ -69,7 +124,7 @@ jobs:
         if: ${{ always() }}
         uses: actions/upload-artifact@v2
         with:
-          name: linting-log-file
+          name: linting-logs
           path: |
             lint_log.txt
             lint_results.md