Fix work item exit code for helix tests #45164
Conversation
The helix work items are only supposed to return non-zero if they fail to report tests for some reason.
|
Tagging subscribers to this area: @safern, @ViktorHofer Issue DetailsThe helix work items are only supposed to return non-zero if they fail
|
|
I don't understand the need of this change. Wouldn't this cause a workitem to be stored as "Pass" on Kusto even if there are test failures? What changed? We've been doing this for years and it has been working just fine. |
|
A workitem is supposed to be "Passed" if it ran tests without crashing. This is causing tons of noise in the test results, because the failed work items get reported as a failed test. |
Really? That seems confusing, if I read data and I see a workitem passed, my instinct would be to think there were test failures.
That's unrelated, the constant test failures are infrastructure issues we're trying to fix. Could you elaborate on which test failures you're referring to in case they're not related? |
|
The "WorkItem" status in kusto becomes "failed" if there are test failures, and "bad exit" if the work item exited with non-zero. On top of that, the test failure report here https://dev.azure.com/dnceng/public/_test/analytics?definitionId=686&contextType=build is filled with "work item failures" that don't mean anything because the workitem "fails" if there are test failures. Each of those workitem failures is supposed to be a crash-level failure that needs an investigation, not a "o this test failed" type failure. |
Ah, ok that makes more sense. |
There was a problem hiding this comment.
@ViktorHofer this will need to be considered when moving to using VSTest, I don't know what exit code VSTest returns when a test fails
ghost
locked as resolved and limited conversation to collaborators
The helix work items are only supposed to return non-zero if they fail
to report tests for some reason.