Timing tests are flaky and this is very bad

[Patater]

Description

• Type: Bug
• Priority: Blocker

---

Bug

OS
All

mbed TLS build:
Version: development (and many other versions)

Expected behavior

The CI should run tests that are not flaky, so we can reliably tell that PRs or new code additions do not cause regressions.

Actual behavior

The timing tests (timing_suite) sometimes fail and sometimes pass.

What we should do about this

CI should run only non-flaky tests. We've made attempts in the past at improving the tests (like in #1136), but we should instead take a step back and instead of just making the timing tests more robust, think about why we have them in the first place. Then we should determine the most sensible course forward to solving the issues that the timing tests were originally written to solve. If this means we wanted to increase our test coverage via the timing tests, we should consider less flaky ways of increasing test coverage. If we wanted some random timing variation testing, we can exclude such exploratory testing from the CI and perform the testing another way.

[ciarmcom]

ARM Internal Ref: IOTSSL-2203

[mpg]

Note: most of the failures I've seen were in test_suite_timing, but some also happen in programs/test/selftest, for example https://jenkins-internal.mbed.com/job/mbedtls-pr/1187/execution/node/80/log/