Overhaul GHA #1200

adamrtalbot · 2023-08-31T09:12:07Z

Streamline Github Actions a bit to make them more efficient.

Tidy workflows into one testing workflow
Tidy up AWS tests into single workflow
Use native cacheing of actions for Python and Nextflow instead of manual cacheing
Match triggers for nf-core template in ci.yml
Use standard ci.yml template which should help with migration to nf-test
Add default tag to pytest-workflow if anything changes within the repo. Always runs the basic 4 tests.

PR checklist

github-actions · 2023-08-31T09:15:53Z

`nf-core lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit 9342569

+| ✅ 142 tests passed       |+
#| ❔   8 tests were ignored |#
!| ❗   4 tests had warnings |!

❗ Test warnings:

files_exist - File not found: .github/workflows/awsfulltest.yml
nextflow_config - Config manifest.version should end in dev: 3.3.0
schema_lint - Input mimetype is missing or empty
schema_description - No description provided in schema for parameter: input_restart

❔ Tests ignored:

files_exist - File is ignored: conf/modules.config
files_unchanged - File ignored due to lint config: assets/nf-core-sarek_logo_light.png
files_unchanged - File ignored due to lint config: docs/images/nf-core-sarek_logo_light.png
files_unchanged - File ignored due to lint config: docs/images/nf-core-sarek_logo_dark.png
files_unchanged - File ignored due to lint config: lib/NfcoreTemplate.groovy
template_strings - template_strings
schema_params - schema_params
multiqc_config - multiqc_config

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: assets/nf-core-sarek_logo_light.png
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/images/nf-core-sarek_logo_light.png
files_exist - File found: docs/images/nf-core-sarek_logo_dark.png
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: lib/nfcore_external_java_deps.jar
files_exist - File found: lib/NfcoreTemplate.groovy
files_exist - File found: lib/Utils.groovy
files_exist - File found: lib/WorkflowMain.groovy
files_exist - File found: main.nf
files_exist - File found: assets/multiqc_config.yml
files_exist - File found: conf/base.config
files_exist - File found: conf/igenomes.config
files_exist - File found: .github/workflows/awstest.yml
files_exist - File found: lib/WorkflowSarek.groovy
files_exist - File found: modules.json
files_exist - File found: pyproject.toml
files_exist - File not found check: Singularity
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: docs/images/nf-core-sarek_logo.png
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: .travis.yml
nextflow_config - Config variable found: manifest.name
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: manifest.homePage
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: params.validationShowHiddenParams
nextflow_config - Config variable found: params.validationSchemaIgnoreParams
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config manifest.name began with nf-core/
nextflow_config - Config variable manifest.homePage began with https://github.com/nf-core/
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - CODE_OF_CONDUCT.md matches the template
files_unchanged - LICENSE matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/CONTRIBUTING.md matches the template
files_unchanged - .github/ISSUE_TEMPLATE/bug_report.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/config.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/PULL_REQUEST_TEMPLATE.md matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - .github/workflows/linting.yml matches the template
files_unchanged - assets/email_template.html matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - assets/sendmail_template.txt matches the template
files_unchanged - docs/README.md matches the template
files_unchanged - lib/nfcore_external_java_deps.jar matches the template
files_unchanged - .gitignore matches the template
files_unchanged - .prettierignore matches the template
files_unchanged - pyproject.toml matches the template
actions_ci - '.github/workflows/ci.yml' is triggered on expected events
actions_ci - '.github/workflows/ci.yml' checks minimum NF version
actions_awstest - '.github/workflows/awstest.yml' is triggered correctly
readme - README Nextflow minimum version badge matched config. Badge: 23.04.0, Config: 23.04.0
readme - README Zenodo placeholder was replaced with DOI.
pipeline_todos - No TODO strings found
pipeline_name_conventions - Name adheres to nf-core convention
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
system_exit - No System.exit calls found
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
actions_schema_validation - Workflow validation passed: clean-up.yml
actions_schema_validation - Workflow validation passed: awstest.yml
actions_schema_validation - Workflow validation passed: fix-linting.yml
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: branch.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'

Run details

nf-core/tools version 2.9
Run at 2023-09-01 11:47:36

asp8200 · 2023-08-31T09:18:47Z

.github/workflows/pytest-workflow.yml

@@ -67,7 +53,6 @@ jobs:
    env:
      NXF_ANSI_LOG: false
      TEST_DATA_BASE: "${{ github.workspace }}/test-datasets"
-      SENTIEON_LICENSE_BASE64: ${{ secrets.SENTIEON_LICENSE_BASE64 }}


Hmmm ... Are you sure about that?

I copied it from the existing workflow, what should it be?

What do you mean you copied it from the existing workflow?

Anyways, I don't think the line should be deleted.

It came from pytest-workflow_release.yml, which I merged with this one. Will add it back!

adamrtalbot · 2023-08-31T09:24:37Z

@nf-core-bot fix linting

maxulysse

Lgtm

maxulysse · 2023-08-31T11:32:23Z

Can you update changelog ?

adamrtalbot · 2023-08-31T11:36:30Z

Can you update changelog ?

Yup, I was waiting until I knew what I actually had changed!

maxulysse · 2023-08-31T12:24:10Z

too many tests are triggered, can you restore the check changes part of the actions?

This reverts commit d1b3f4e.

adamrtalbot · 2023-08-31T14:59:50Z

too many tests are triggered, can you restore the check changes part of the actions?

Done, but we need a way of avoiding the paths-filter hitting the API limit. Hmmmmmmm.

adamrtalbot · 2023-08-31T15:27:07Z

@maxulysse what does this parameter do and is it the cause of our current problems?

sarek/conf/test/cache.config

Line 122 in 2454384

params.singularity_pull_docker_container = false

Error:

ERROR ~ JSONObject["singularity_pull_docker_container"] not found.

maxulysse · 2023-08-31T15:33:03Z

Good catch

maxulysse · 2023-08-31T15:35:26Z

Good catch

never mind, we don't that in dev anymore...

adamrtalbot · 2023-08-31T16:03:29Z

Good catch

never mind, we don't that in dev anymore...

Fixed with #1203

maxulysse · 2023-08-31T16:13:26Z

Good catch

never mind, we don't that in dev anymore...

yeah was looking at the wrong branch...

maxulysse · 2023-08-31T16:30:03Z

add .git/workflows/ci.yml to the .nf-core.yml in files_exist

.github/workflows/pytest-workflow.yml

…arlier)

edmundmiller

Looks great to me!

Do you have an issue for the file-paths hitting the API rate limit? Maybe we should open it with them?

FriederikeHanssen

some of it is a black box to me, but from what i see it is streamlining the tests, not changing them, right?

FriederikeHanssen · 2023-09-01T11:05:37Z

.github/workflows/awstest.yml

@@ -4,10 +4,24 @@ name: nf-core AWS test

 on:
  workflow_dispatch:
+    inputs:


nice I like it, much cleaner to have it all here

adamrtalbot · 2023-09-01T11:09:47Z

Looks great to me!

Do you have an issue for the file-paths hitting the API rate limit? Maybe we should open it with them?

Yes we do. I tried to fix by using an empty token ('') so it used git directly but it just thought everything was different and ran 1 zillion tests. Quite a few people have hit the limit on their Github issues page.

adamrtalbot · 2023-09-01T11:11:22Z

some of it is a black box to me, but from what i see it is streamlining the tests, not changing them, right?

yup, condensing multiple workflows to single files wherever possible. Removing older cruft. I may have introduced some problems once it's on master but it's quite hard to test before merging.

FriederikeHanssen · 2023-09-01T11:15:16Z

sorry didn't mean to edit your comment but quote as reply. 🤦‍♀️ goind to get some coffee ☕

FriederikeHanssen · 2023-09-01T11:16:48Z

some of it is a black box to me, but from what i see it is streamlining the tests, not changing them, right?

yup, condensing multiple workflows to single files wherever possible. Removing older cruft. I may have introduced some problems once it's on master but it's quite hard to test before merging.

ok then let's merge and be ready for a patch release

maxulysse

LGTM.
Anything we can push to TEMPLATE as well?

adamrtalbot · 2023-09-01T11:58:29Z

LGTM.
Anything we can push to TEMPLATE as well?

Kinda, if you look it only includes a very trivial example of running nextflow run . -profile test: https://github.com/nf-core/tools/blob/e5ce6ce20304835bd40f102f038b7e1aadc888b2/nf_core/pipeline-template/.github/workflows/ci.yml

So it's a little hard to migrate the changes over. Will we be adding pytest-workflow and the change tracking to them? Maybe we should? Perhaps it's worth a ticket and discussion before adding new code?

I can merge the AWS tests but that's the smallest part of this PR.

adamrtalbot requested a review from maxulysse as a code owner August 31, 2023 09:12

asp8200 reviewed Aug 31, 2023

View reviewed changes

maxulysse approved these changes Aug 31, 2023

View reviewed changes

adamrtalbot and others added 14 commits August 31, 2023 15:54

Tidy workflows into one testing workflow

3957e7a

Tidy up AWS tests into single workflow

39e4f89

Condense linting workflows to single workflow

5567e32

Add confirm-pass check to the end of the testing workflow

6438b68

[automated] Fix linting with Prettier

83d81ea

Add Sentieon secret back in

a1436a9

Fix tags for alignment checks

754684f

Add default test to list of existing tags

77e800f

Add merge group in case of a merge queue

5f341e2

Setup Cache for Nextflow and Pip

52bc779

Revert "Condense linting workflows to single workflow"

204950a

This reverts commit d1b3f4e.

unique-ify the installation cache key

eaac9e5

Use git commands instead of GHA CLI to check for path differences

c67d751

Use explicit requirements.txt file

a64e540

adamrtalbot force-pushed the overhaul_ci_gha branch from 6d39549 to a64e540 Compare August 31, 2023 14:57

Remove blank token from paths-filter

1cced6d

lint requirements.txt

2454384

Merge branch 'dev' of github.com:nf-core/sarek into overhaul_ci_gha

a43de5a

adamrtalbot commented Sep 1, 2023

View reviewed changes

.github/workflows/pytest-workflow.yml Outdated Show resolved Hide resolved

adamrtalbot added 7 commits September 1, 2023 09:51

Update .github/workflows/pytest-workflow.yml

827ef19

Match CI triggers to nf-core schema

15ec813

CHANGELOG

84dc775

Rely on setup-python auto cacheing feature instead of manual cacheing

8f61be2

fixup

439d837

fixup

855c878

Change order to install test data after software dependencies (fail e…

d804599

…arlier)

adamrtalbot requested a review from maxulysse September 1, 2023 09:51

edmundmiller approved these changes Sep 1, 2023

View reviewed changes

FriederikeHanssen reviewed Sep 1, 2023

View reviewed changes

FriederikeHanssen approved these changes Sep 1, 2023

View reviewed changes

adamrtalbot added 5 commits September 1, 2023 12:17

Change triggers for AWS tests

919115f

Change triggers for AWS tests

b40397c

Change triggers for AWS tests

2866617

Change triggers for AWS tests

aa9e189

Change triggers for AWS tests

9342569

maxulysse reviewed Sep 1, 2023

View reviewed changes

maxulysse approved these changes Sep 1, 2023

View reviewed changes

adamrtalbot merged commit 43a8704 into dev Sep 1, 2023
16 checks passed

adamrtalbot deleted the overhaul_ci_gha branch September 1, 2023 15:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Overhaul GHA #1200

Overhaul GHA #1200

adamrtalbot commented Aug 31, 2023 •

edited

Loading

github-actions bot commented Aug 31, 2023 •

edited

Loading

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

asp8200 Aug 31, 2023

adamrtalbot Aug 31, 2023

asp8200 Aug 31, 2023

adamrtalbot Aug 31, 2023

adamrtalbot commented Aug 31, 2023

maxulysse left a comment

maxulysse commented Aug 31, 2023

adamrtalbot commented Aug 31, 2023

maxulysse commented Aug 31, 2023

adamrtalbot commented Aug 31, 2023

adamrtalbot commented Aug 31, 2023

maxulysse commented Aug 31, 2023

maxulysse commented Aug 31, 2023

adamrtalbot commented Aug 31, 2023

maxulysse commented Aug 31, 2023

maxulysse commented Aug 31, 2023

edmundmiller left a comment

FriederikeHanssen left a comment

FriederikeHanssen Sep 1, 2023

adamrtalbot commented Sep 1, 2023

adamrtalbot commented Sep 1, 2023 •

edited by FriederikeHanssen

Loading

FriederikeHanssen commented Sep 1, 2023

FriederikeHanssen commented Sep 1, 2023

maxulysse left a comment

adamrtalbot commented Sep 1, 2023 •

edited

Loading

Overhaul GHA #1200

Overhaul GHA #1200

Conversation

adamrtalbot commented Aug 31, 2023 • edited Loading

PR checklist

github-actions bot commented Aug 31, 2023 • edited Loading

nf-core lint overall result: Passed ✅ ⚠️

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

asp8200 Aug 31, 2023

Choose a reason for hiding this comment

adamrtalbot Aug 31, 2023

Choose a reason for hiding this comment

asp8200 Aug 31, 2023

Choose a reason for hiding this comment

adamrtalbot Aug 31, 2023

Choose a reason for hiding this comment

adamrtalbot commented Aug 31, 2023

maxulysse left a comment

Choose a reason for hiding this comment

maxulysse commented Aug 31, 2023

adamrtalbot commented Aug 31, 2023

maxulysse commented Aug 31, 2023

adamrtalbot commented Aug 31, 2023

adamrtalbot commented Aug 31, 2023

maxulysse commented Aug 31, 2023

maxulysse commented Aug 31, 2023

adamrtalbot commented Aug 31, 2023

maxulysse commented Aug 31, 2023

maxulysse commented Aug 31, 2023

edmundmiller left a comment

Choose a reason for hiding this comment

FriederikeHanssen left a comment

Choose a reason for hiding this comment

FriederikeHanssen Sep 1, 2023

Choose a reason for hiding this comment

adamrtalbot commented Sep 1, 2023

adamrtalbot commented Sep 1, 2023 • edited by FriederikeHanssen Loading

FriederikeHanssen commented Sep 1, 2023

FriederikeHanssen commented Sep 1, 2023

maxulysse left a comment

Choose a reason for hiding this comment

adamrtalbot commented Sep 1, 2023 • edited Loading

adamrtalbot commented Aug 31, 2023 •

edited

Loading

github-actions bot commented Aug 31, 2023 •

edited

Loading

`nf-core lint` overall result: Passed ✅ ⚠️

adamrtalbot commented Sep 1, 2023 •

edited by FriederikeHanssen

Loading

adamrtalbot commented Sep 1, 2023 •

edited

Loading