Mark testset as being flaky

[jonas-schulze]

as an alternative to #38211.

[jonas-schulze]

Do you have a procedure for logging and monitoring these kinds of failures over several builds/PRs/... to get an overview about the percentage of failures? E.g. write some file of specific format/location, that is picked up by CI (similar to results of coverage analysis); add success or failure to prometheus when run in CI; or similar. This could be helpful to discover if flaky tests get better (which could be a sign of them having been fixed by accident) or worse.

[vchuravy]

Do you have a procedure for logging and monitoring these kinds of failures over several builds/PRs/... to get an overview about the percentage of failures? E.g. write some file of specific format/location, that is picked up by CI; add success or failure to prometheus when run in CI; or similar. This could be helpful to discover if flaky tests get better (which could be a sign of them having been fixed by accident) or worse.

No we don't, we would need the test set reporting to be understood by our CI system, which right now is buildbot. It seems deeper integration is possible with buildkite which we are looking at to move to. Reporting tests to something like prometheus might be interesting, especially if we also get test duration.

cc: @staticfloat @christopher-dG

[christopher-dG]

Buildkite has by default a "reliability" metric that is basically a pass/fail ratio of entire jobs, but to get per-testset resolution we'd need to do a custom plugin. Seems doable though.

[Keno]

We shouldn't run tests that have unknown failure modes. If they introduce memory or other corruption, we could still be tracking down mystery failures.

[vtjnash]

This is semi-isolated in that, while the race conditions it tests definitely have been seen to corrupt memory, they are newly minted addprocs machines and so probably won't bring down the master node.

[Keno]

Do we trust the error handling in Distributed sufficiently that arbitrary memory corruption in the worker node will not cause strange failures on the master node?

[vtjnash]

I don't trust anyone

[jonas-schulze]

I guess this PR isn't needed anymore. Shall the discussion about monitoring test set failures be documented elsewhere?

[musm]

I guess this PR isn't needed anymore. Shall the discussion about monitoring test set failures be documented elsewhere?

I think our current idea is to report them under the CI tag (if you don't find an issue corresponding to the failure you see in CI, please open one up).