Feedback about Test Analytics and Flaky Test Reporting

[rohan-at-sentry]

Thanks for dropping by! 👋

We've recently released a whole new feature around Test analytics and are working on reporting Test Flakes on your PR ❄️ .

We'd love to hear feedback on

• How your setup experience was.
• How easy/useful the PR comment is

This issue is intended to share and collect feedback about the tool. If you have support needs or questions, please let us know!

[houserx-jmcc]

• How your setup experience was.
  • Very easy, worked on the first go after adding the new GitHub action step.
• How easy/useful the PR comment is
  • I like that it is combined with the existing coverage comment versus being a separate one.

In the event this action is not used in the context of a pull request (i.e. a scheduled test, pre-release test, etc.), it would be nice to be able to also display the results in GitHub's Job Summary in lieu of a PR comment. We currently use https://github.com/phoenix-actions/test-reporting to do this and it works nicely. When sharing test results, permalinking to the job summary works better due to the bigger screen size it can occupy. If this feature was added, we'd switch over completely!

[houserx-jmcc]

An update: we're also seeing the below error intermittently that we have been unfortunately seeing in v4 of the standard codecov-action (codecov/codecov-action#1280):

Error: write EPIPE
    at afterWriteDispatched (node:internal/stream_base_commons:160:15)
    at writeGeneric (node:internal/stream_base_commons:151:3)
    at Socket._writeGeneric (node:net:952:11)
    at Socket._write (node:net:964:8)
    at writeOrBuffer (node:internal/streams/writable:447:12)
    at _write (node:internal/streams/writable:389:10)
    at Socket.Writable.end (node:internal/streams/writable:665:17)
    at Socket.end (node:net:7[22](https://github.com/houserx/platform/actions/runs/8513226507/job/23316490519?pr=8922#step:12:23):31)
    at module.exports (/runner/_work/_actions/codecov/test-results-action/v1/node_modules/gpg/lib/spawnGPG.js:50:1)
    at Object.call (/runner/_work/_actions/codecov/test-results-action/v1/node_modules/gpg/lib/gpg.js:28:1)

[rohan-at-sentry]

Thanks for your feedback @houserx-jmcc - we're looking into adding support for reporting on GH Job summary as well

Re the issue with EPIPE - yeah it's something we've noticed. We're still working on a fix for this

[rohan-at-sentry]

@houserx-jmcc out of curiosity what non PR usecases are you using https://github.com/phoenix-actions/test-reporting for?

Off the top of my head, I'd guess something along the lines of running tests before deploy (of a production image/release) etc. Are there other usecases that you can share?

[houserx-jmcc]

Off the top of my head, I'd guess something along the lines of running tests before deploy (of a production image/release) etc. Are there other usecases that you can share?

Yup! That is one. Others include: running the tests on a stable branch while investigating suspected flakey tests, browser tests we run after deployments to increase confidence, and scheduled tests that periodically perform data quality checks against our stable environment databases. We use Jest for all these use cases and thus can easily pipe the XML into the test action reporter.

And for viewing historical test results on a pull request, the job summary is tied to each workflow execution. The PR comment is auto-updated though, so checking on past results becomes a bit more obtuse.

[rohan-at-sentry]

@houserx-jmcc - is this directionally what you had in mind? (still early days!)

https://github.com/joseph-sentry/codecov-cli/actions/runs/8560172298

[houserx-jmcc]

Precisely! Looks great :)

[rohan-at-sentry]

@houserx-jmcc - We're currently testing a new version of our test analytics feature internally where we're able to detect and report on Flaky Tests. At this time, I'm reaching out to everyone who has set up Codecov Test Analytics to invite feedback on the UI of our Flaky Test Feature. If this is of interest to you, please let me know and I'll reach out with some scheduling options.

[houserx-jmcc]

@rohan-at-sentry Happy to provide some additional feedback 👍

[rohan-at-sentry]

thanks @houserx-jmcc. Here are some times that work for us (you can find time beginning next week if that works better) - https://calendar.app.google/7qTT3zeshUnrEHMy5

cc @Adal3n3

[Blacksmoke16]

I read through the docs a few times and I have a question. For context, I have a monorepo that I plan on setting up with codecov via the components feature where by I'll run the specs/generate a report for each component into a dedicated directory that'll then be uploaded all at once via the official GHA which will handle assigning each report to the proper component.

However, what I'd like to confirm is would doing something similar for the junit reports work in a similar way? Where codecov will just know which component to assign the test results to? Does this feature even work with components at the moment? Or is it global to a repo and long as I upload all the junit reports it'll just work?

[rohan-at-sentry]

@Blacksmoke16 currently yes the following is an accurate way to describe the system

global to a repo and long as I upload all the junit reports it'll just work

For clarity, right now we'd simply output these results onto the tests tab. You can see this in action on our own repos here
Further -

codecov will just know which component to assign the test results to?

is not supported currently, but I'd like to learn more

We're thinking of ways to group around test suites and environments (those are things we've heard from customers we're working with), but I'd be curious to hear what other groupings are top of mind for you. Also, do you envision these "groupings" to be identical to the components you'd set up for coverage? If so, why?

[Blacksmoke16]

For clarity, right now we'd simply output these results onto the tests tab.

Ah, okay that example helps a lot. Basically is just a big list of all tests.

Also, do you envision these "groupings" to be identical to the components you'd set up for coverage? If so, why?

Uhh, I'm still in the reading/research/thinking phase of this as I have yet to actually get reports uploading or anything so I don't have a ton of useful insight just yet. However my initial reasoning was like, in my case at least, a component is a individual independent project that just happens to be in a monorepo. So it just made sense that tests output would be tied to a specific component to more easily see stats in that regard versus all components at once. Then you could more easily answer like "what is the most failure prone component?" Then, as you pointed out, flags would still make sense for filtering purposes to maybe spot issues like "our integration tests in staging have a lot higher failure rate than on prod" or something along those lines.

But honestly it could also make sense to have the tests tab be more like higher level stats/aggregates and the actual test results be more tightly coupled to a specific commit/pr? I'll be sure to report back if I think of anything else after actually getting things setup and playing with it more.

[webknjaz]

@rohan-at-sentry in one repo, test uploads crash fail with Upload failed: {"service":["This field may not be null."]} on GHA but not locally: codecov/codecov-cli#486 / https://github.com/ansible/awx-plugins/actions/runs/10581340011/job/29318439675#step:23:47. I haven't been able to identify the exact cause. My understanding is that the confusing error is coming from Django in https://github.com/codecov/codecov-api. It would be useful to both improve logging and fix the issue in the uploader.

[rohan-at-sentry]

Thanks, @webknjaz I opened codecov/engineering-team#2437 to assess

[kevinphamsigma]

thanks @houserx-jmcc. Here are some times that work for us (you can find time beginning next week if that works better) - https://calendar.app.google/7qTT3zeshUnrEHMy5

cc @Adal3n3

Is the flaky test detection enabled automatically at this point? Or something we need to enable? Also happpy to provide some feedback. Seems like this has good potential but missing some key features that would make it more useful for us

[rohan-at-sentry]

@kevinphamsigma yeah flaky test detection should be enabled at this point. I'd love to hear feedback. Can I get you to book some time with us using this link

[jluebbe]

The JUnit XML files generated by meson are not parsed: jluebbe/rauc#4 (comment)

In the web dashboard tab for tests, "No test results found for this branch" is shown.

[undsoft]

I would suggest making it more explicit that test-results-action does not replace codecov/codecov-action.
It's not immediately obvious from instructions on Tests page in CI and github page for test-results-action doesn't specify its purpose. You have to scroll down to final example to find it.

[fmigneault]

After enabling this feature to try it out, the very detailed coverage report I was previously getting is now replaced by an irrelevant message (see below) that tells me 4 tests failed. Note that, in the case of that PR (crim-ca/weaver#758), all tests succeeded.

Is that expected behavior or a bug? Both actions are simultaneously enabled in the CI as per the example documentation

I would expect failed tests to be reported only when they failed. So where does these failures come from?

Previous Detailed Coverage Report (before Test Analytics was enabled)

crim-ca/weaver#756 (comment)

Irrelevant Test Results?

crim-ca/weaver#758 (comment)