Flaky tests: Explore using skip instead of dontcare

[jbergstroem]

While running the test suite, passing dontcare to --flaky-tests will make the runner run a test but don't care about the output. In a few cases we've marked tests as flaky that times out. This means that we (when not using -J) will sit and wait for a minute.

One counter argument would be that these tests could start failing for other reasons, but I still think the net gain is bigger. At the end of the day, we should encourage debugging and fixing flaky tests. Thoughts?

[joaocgreis]

The yellow ball of dontcare is more of a reminder that the test is still flaky, can't think of more reasons to keep dontcare. So, +1

[orangemocha]

I am a bit concerned that we'd start losing visibility on the failing tests. What if a flaky test starts failing consistently, for a different reason?

I guess I am +0, but let some job (node-daily-master) not ignore flaky tests.

[rvagg]

I want yellow balls to show up on flaky failures, I really don't want the fact that we have this "flaky" thing in here lose visibility, I want collaborators OCD to kick in when they don't get all-green and make them go hunting and solving flakies, so I'm -1 if this means losing the yellow

[Fishrock123]

If this makes show up as green: nope nope nope

[orangemocha]

I also agree it's better to continue running them.

[jbergstroem]

Ok, I believe we have somewhat of a consensus then. We'll keep running flaky tests.

TL;DR

Pro:

• avoid having flaky tests interfer with others
• speed up test suite since flaky tests still wait for timeout

Con:

• increased visibility of a fail
• better test coverage
• encourage fixing a fail instead of moving to next