Flaky tests

[na--]

Just a list of tests we have noticed are flaky in CI runs, so we don't forget to fix/refactor them before merging #1007 in master:

TestVariableLoopingVUsRun - https://ci.appveyor.com/project/robingustafsson/k6/builds/31408611

[na--]

TestVariableLoopingVUsRampDownNoWobble - https://circleci.com/gh/loadimpact/k6/9110

[imiric]

I refactored these two tests and they seem more stable, but while stress testing lib/executor, I've also seen these fail occasionally (mostly on Windows; ordered by approx. failure rate, highest first):

• TestExternallyControlledRun
• TestConstantArrivalRateRunCorrectRate
• TestVariableArrivalRateRunCorrectRate
• TestConstantLoopingVUsRun
• TestPerVUIterationsRunVariableVU
• TestSharedIterationsRunVariableVU

[mstoykov]

they will probably get changed with the striping algorithm and rewrite so maybe just wait until then ?

[imiric]

Another one 😞: core.TestMetricsEmission: https://circleci.com/gh/loadimpact/k6/9243

[na--]

Another TestConstantArrivalRateRunCorrectRate failure: https://circleci.com/gh/loadimpact/k6/9464

[na--]

TestExternallyControlledRun flakiness: https://circleci.com/gh/loadimpact/k6/9530

[na--]

Food for thought: https://dmitryfrank.com/articles/mocking_time_in_go (from https://news.ycombinator.com/item?id=23292215)

[imiric]

I'm exploring that article and the time mocking packages, but one thing concerns me, mentioned in golang/go#8869:

The benbjohnson clock package attempts to make Ticker and Timer calls (as well as their
related AfterFunc and Sleep methods) work against a fake time that can be set and
updated in tests. However, it relies on some micro-sleeps and runtime.Gosched calls that
are obviously going to be flaky.

It would be a substantial effort to convert the entire codebase to use this Clock interface. Is that really worth it if it doesn't guarantee deterministic tests?

Also see grafana/metrictank#342, where they abandon it altogether.

The article author uses a fork where this might not be an issue(?), but not sure if this can be fixed outside of the stdlib.
So I think the way forward will be much harder: rewriting tests to ensure they rely on timing as little as possible or not at all, which might be impossible in some cases. I'll take a look at some of the current ones and see what can be done.

[imiric]

A recently added one: TestConstantArrivalRateDroppedIterations, seems to fail frequently:

https://ci.appveyor.com/project/robingustafsson/k6/builds/33999101
https://ci.appveyor.com/project/robingustafsson/k6/builds/33998965

[na--]

I'm closing this, since most, if not all, of these tests should have been fixed by now. We can make new issues for every new flaky test we find.