-
Notifications
You must be signed in to change notification settings - Fork 6
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feedback about Test Analytics and Flaky Test Reporting #304
Comments
In the event this action is not used in the context of a pull request (i.e. a scheduled test, pre-release test, etc.), it would be nice to be able to also display the results in GitHub's Job Summary in lieu of a PR comment. We currently use https://github.com/phoenix-actions/test-reporting to do this and it works nicely. When sharing test results, permalinking to the job summary works better due to the bigger screen size it can occupy. If this feature was added, we'd switch over completely! |
An update: we're also seeing the below error intermittently that we have been unfortunately seeing in v4 of the standard codecov-action (codecov/codecov-action#1280):
|
Thanks for your feedback @houserx-jmcc - we're looking into adding support for reporting on GH Job summary as well Re the issue with EPIPE - yeah it's something we've noticed. We're still working on a fix for this |
@houserx-jmcc out of curiosity what non PR usecases are you using https://github.com/phoenix-actions/test-reporting for? Off the top of my head, I'd guess something along the lines of running tests before deploy (of a production image/release) etc. Are there other usecases that you can share? |
Yup! That is one. Others include: running the tests on a stable branch while investigating suspected flakey tests, browser tests we run after deployments to increase confidence, and scheduled tests that periodically perform data quality checks against our stable environment databases. We use Jest for all these use cases and thus can easily pipe the XML into the test action reporter. And for viewing historical test results on a pull request, the job summary is tied to each workflow execution. The PR comment is auto-updated though, so checking on past results becomes a bit more obtuse. |
@houserx-jmcc - is this directionally what you had in mind? (still early days!) https://github.com/joseph-sentry/codecov-cli/actions/runs/8560172298 |
Precisely! Looks great :) |
@houserx-jmcc - We're currently testing a new version of our test analytics feature internally where we're able to detect and report on Flaky Tests. At this time, I'm reaching out to everyone who has set up Codecov Test Analytics to invite feedback on the UI of our Flaky Test Feature. If this is of interest to you, please let me know and I'll reach out with some scheduling options. |
@rohan-at-sentry Happy to provide some additional feedback 👍 |
thanks @houserx-jmcc. Here are some times that work for us (you can find time beginning next week if that works better) - https://calendar.app.google/7qTT3zeshUnrEHMy5 cc @Adal3n3 |
I read through the docs a few times and I have a question. For context, I have a monorepo that I plan on setting up with codecov via the components feature where by I'll run the specs/generate a report for each component into a dedicated directory that'll then be uploaded all at once via the official GHA which will handle assigning each report to the proper component. However, what I'd like to confirm is would doing something similar for the junit reports work in a similar way? Where codecov will just know which component to assign the test results to? Does this feature even work with components at the moment? Or is it global to a repo and long as I upload all the junit reports it'll just work? |
@Blacksmoke16 currently yes the following is an accurate way to describe the system
For clarity, right now we'd simply output these results onto the tests tab. You can see this in action on our own repos here
is not supported currently, but I'd like to learn more We're thinking of ways to group around test suites and environments (those are things we've heard from customers we're working with), but I'd be curious to hear what other groupings are top of mind for you. Also, do you envision these "groupings" to be identical to the components you'd set up for coverage? If so, why? |
Ah, okay that example helps a lot. Basically is just a big list of all tests.
Uhh, I'm still in the reading/research/thinking phase of this as I have yet to actually get reports uploading or anything so I don't have a ton of useful insight just yet. However my initial reasoning was like, in my case at least, a component is a individual independent project that just happens to be in a monorepo. So it just made sense that tests output would be tied to a specific component to more easily see stats in that regard versus all components at once. Then you could more easily answer like "what is the most failure prone component?" Then, as you pointed out, flags would still make sense for filtering purposes to maybe spot issues like "our integration tests in staging have a lot higher failure rate than on prod" or something along those lines. But honestly it could also make sense to have the tests tab be more like higher level stats/aggregates and the actual test results be more tightly coupled to a specific commit/pr? I'll be sure to report back if I think of anything else after actually getting things setup and playing with it more. |
Thanks for dropping by! 👋
We've recently released a whole new feature around Test analytics and are working on reporting Test Flakes on your PR ❄️ .
We'd love to hear feedback on
This issue is intended to share and collect feedback about the tool. If you have support needs or questions, please let us know!
The text was updated successfully, but these errors were encountered: