Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The return of daily tests #10320

Merged
merged 1 commit into from
Jul 3, 2024
Merged

Conversation

chris48s
Copy link
Member

@chris48s chris48s commented Jul 2, 2024

Refs #8821

This PR adds a GitHub Workflow that runs our tests every day and spits out a markdown summary.

Here's an example run: https://github.com/badges/shields/actions/runs/9749868442

This doesn't completely replicate all of the functionality of the old daily tests repo. That repo also measured and reported code coverage. However, I encountered two problems implementing this:

  1. Both the markdown summary and coverage reports are implemented as mocha reporters. Out of the box, it is not possible to tell mocha to use 2 reporters at once, but there may be a way to work round this. I do want to produce human readable output as well as coverage XML though.
  2. There's an issue with c8 that causes builds to fail when measuring coverage on Node 20. There's an unmerged upstream PR at fix: prevent crash when v8 incorrectly merges static_initializer's bcoe/v8-coverage#2

I think both of those problems can be solved in time:

  • I can work on finding a way round the reporters issue.
  • The c8 bugfix may be merged upstream. If not, eventually node 22 (which doesn't have this issue) will be our target platform.

That said, right now I have a working PR for an action which runs all the tests every day and outputs a summary we can read. That's more than we have now, so I suggest lets do this as step one rather than blocking on those things. Lets not let perfect be the enemy of good. Regaining visibility on what is failing will help us start fixing stuff.

@chris48s chris48s added the developer-experience Dev tooling, test framework, and CI label Jul 2, 2024
Copy link
Contributor

github-actions bot commented Jul 2, 2024

Messages
📖 ✨ Thanks for your contribution to Shields, @chris48s!

Generated by 🚫 dangerJS against 86c1e22

Copy link
Member

@PyvesB PyvesB left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for putting this together, looks good to me! 👍🏻

@chris48s chris48s added this pull request to the merge queue Jul 3, 2024
Merged via the queue into badges:master with commit e4a9c73 Jul 3, 2024
24 checks passed
@chris48s chris48s deleted the daily-tests-gha2 branch July 3, 2024 17:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
developer-experience Dev tooling, test framework, and CI
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants