flaky test failure in RegistryDelete

[rade]

I ran into this yesterday in a local build - re-running "fixed it" - and @bboreham encountered it in CI today

--- FAIL: TestRegistryDelete (0.01s)
	registry_test.go:511: Didn't get right container updates: 
		--- want
		+++ have
		@@ -1,4 +1,322 @@
		-([]report.Node) (len=1 cap=1) {
		+([]report.Node) (len=2 cap=2) {
                ...

Presumably a race condition in the test rather than a bug in the code.

[satyamz]

I am also facing this issue on local make tests with -slow flag.

[bboreham]

Looks the same here: https://circleci.com/gh/weaveworks/scope/8977

I suspect the 100ms timeout.

[bboreham]

I suspect the 100ms timeout.

except I just raised it to 10 seconds and it still fails.

[bboreham]

The race is:

• newMockClient() sets up a fake Docker with one container, container1.
• TestRegistryDelete() calls testRegistry() calls NewRegistry() which starts a goroutine that calls listenForEvents()
• TestRegistryDelete() calls registry.WatchContainerUpdates()
• If listenForEvents() runs after registry.WatchContainerUpdates(), then the callback is fired with info about container1.
• If listenForEvents() runs before registry.WatchContainerUpdates(), then the callback is not fired.

The test fails in the first case because it has two updates instead of the required one.

[bboreham]

Reopening: more TestRegistryDelete test failures recently:
https://circleci.com/gh/weaveworks/scope/12848
https://circleci.com/gh/weaveworks/scope/12880
https://circleci.com/gh/weaveworks/scope/12884
https://circleci.com/gh/weaveworks/scope/12885