Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Workflow validation #820

Merged
merged 10 commits into from
Jan 7, 2021
Merged

Workflow validation #820

merged 10 commits into from
Jan 7, 2021

Conversation

KevinMenden
Copy link
Contributor

PR checklist

  • This comment contains a description of changes (with reason)
  • CHANGELOG.md is updated
  • If you've fixed a bug or added code that should be tested, add tests!
  • Documentation in docs is updated

Added the lint function actions_schema_validation which uses the GitHub workflow JSON schema to validate GitHub action workflows in .github/workflows/.

As suggested in #795

@codecov
Copy link

codecov bot commented Jan 5, 2021

Codecov Report

Merging #820 (70eda63) into dev (fe15509) will decrease coverage by 0.05%.
The diff coverage is 50.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##              dev     #820      +/-   ##
==========================================
- Coverage   78.40%   78.34%   -0.06%     
==========================================
  Files          22       22              
  Lines        2477     2480       +3     
==========================================
+ Hits         1942     1943       +1     
- Misses        535      537       +2     
Impacted Files Coverage Δ
nf_core/lint/schema_lint.py 80.95% <0.00%> (ø)
nf_core/lint/actions_lint.py 75.60% <60.00%> (-3.34%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 14f54a3...70eda63. Read the comment docs.

Copy link
Member

@ewels ewels left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love it! Super clean code, just as I'd hoped 🚀

Would be nice to add the ability to ignore specific files if listed in the linting config. Pattern for doing this in some other lint tests somewhere.

nf_core/lint/actions_schema_validation.py Outdated Show resolved Hide resolved
nf_core/lint/actions_schema_validation.py Outdated Show resolved Hide resolved
# load workflow
wf_path = os.path.join(self.wf_path, ".github/workflows", wf)
with open(wf_path, "r") as fh:
wf_json = yaml.safe_load(fh)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be good to catch errors raised from yaml syntax problems and log as a failure here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point - those files are also loaded in the other actions_* lint checks, I guess it makes sense to catch those errors there as well? Could basically just copy-paste the same code. We'll have a few redundant errors reported then, if the yml is wrong.

nf_core/lint/actions_schema_validation.py Outdated Show resolved Hide resolved
@KevinMenden
Copy link
Contributor Author

Thanks for the review!
Sure, should be easy to add to the code.

Copy link
Member

@ewels ewels left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love it! Haven't tested but review on my phone looks good. Couple of very minor suggestions but LGTM ✅

nf_core/lint/actions_schema_validation.py Outdated Show resolved Hide resolved
Validate workflows against a schema

Uses the JSON Schema from
https://json.schemastore.org/github-workflow
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs to be valid RST to make the hyperlink work in the docs.

Would also be good to add a bit more detail here - which files exactly? And where to get help (the URL is just a big JSON document). Have to assume that people reading this might not know that these files even exist..

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aaah sorry, didn't think about that this will be rendered at that moment 😱
Will write a nicer message, thanks!

@KevinMenden
Copy link
Contributor Author

Added a longer description - not sure if there's anything else worth mentioning there except what these workflow files are for, where they are, and what the lint test checks. Let me know if something else comes to your mind!

Also added a commit to cast the exception at the schema validation to string, which should make the code more robust (as mentioned in slack).

Copy link
Member

@ewels ewels left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perfect ✨

@ewels ewels merged commit bb5c10d into nf-core:dev Jan 7, 2021
@KevinMenden KevinMenden deleted the workflow-validation branch January 18, 2021 11:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants