file: migrate batch test to non flaky harness #3584 #5715
Conversation
|
Is it possible to get the remaining tests reworked before we release 3.8.0? It'd be good to enable them in the platform again. They've been disabled there for 2 years... |
|
@jamesnetherton Yes this file test flakiness is a long story. Given there was multiple weaknesses and even suspicion of multiple tests polluting each other, the adopted strategy so far is to migrate test by test. That way we have time to check whether a flakiness is re-included at each step. It's an endurance race and we should reach the end soon, not sure for 3.8 with the current strategy applied. Given that, is it still preferred that we migrate everything at once ? The worst case is we reintroduce one of the flakiness back and need to restart investigation from there. In any case, probably we would like to see a month of full file tests without flakiness before re-enabling on the platform ? |
We could do that. Although, I don't recall seeing any file related test failures on the CI builds for a long time now. Hence thought we could take an eager approach and reinstate them in the platform. |
|
Indeed, up to now it's carefully verified at each migrated test that none of the ci failures relate to file test flakiness anymore. As far as I see, there are only 2 tests left to migrate. Fingers crossed they can fit in the new test harness without re-introducing flakiness 🤞 . Concerning platform, it's true that there is a possibility to reinstate the tests even if not fully migrated. We would then collect information by both watching main ci builds plus platform build stability. |
No description provided.