Flaky: TestUnhealthyGameServersWithoutFreePorts

[drichardson]

What happened:

TestUnhealthyGameServersWithoutFreePorts failed after an unrelated documentation change commit: 3bef111

--- FAIL: TestUnhealthyGameServersWithoutFreePorts (153.06s)
    gameserver_test.go:202: 
        	Error Trace:	gameserver_test.go:202
        	Error:      	Received unexpected error:
        	            	timed out waiting for the condition
        	            	waiting for GameServer to be Unhealthy default/udp-serverhswrq
        	            	agones.dev/agones/test/e2e/framework.(*Framework).WaitForGameServerState
        	            		/go/src/agones.dev/agones/test/e2e/framework/framework.go:162
        	            	agones.dev/agones/test/e2e.TestUnhealthyGameServersWithoutFreePorts
        	            		/go/src/agones.dev/agones/test/e2e/gameserver_test.go:201
        	            	testing.tRunner
        	            		/usr/local/go/src/testing/testing.go:909
        	            	runtime.goexit
        	            		/usr/local/go/src/runtime/asm_amd64.s:1357
        	Test:       	TestUnhealthyGameServersWithoutFreePorts

What you expected to happen:

Test passes.

How to reproduce it (as minimally and precisely as possible):

Don't know how. It's another flaky e2e test.

Anything else we need to know?:
no

Environment:
From CI using Google Cloud Build

[aLekSer]

One more time it failed here:
https://console.cloud.google.com/cloud-build/builds/b28fc88f-a995-424e-bdc8-1c31547be6f3?project=agones-images

[akremsa]

I tried to run this particular test more than 200 times - no errors.
It seems this test could be reproduced in scope with other tests.

[aLekSer]

First of all this test on gcloud-test-cluster creates 6 GameServers (why should be 4, as gameserver NodePool is of size 4).
In this test we are trying Static PortPolicy with port 7515 from the range available for other GameServers was used (7000-8000). Then test is creating node_count (for range nodes.Items in the test) more GameServers with the same port, after creating one more and wait until it would be Unhealthy. But what would happen if someone has used this port for other GameServer in a separate test?

I assume in this scenario this GameServers could free this port (in parallel test) before node_count+1 GameServer would be created and it could become Ready instead, port now is free, and Unhealthy could be port number node_count - 1 ( any in the first for loop).
We also need to add CreateGameServerAndWaitUntilReady() for all first node_count gameservers.

So there are two solutions:

1. Use Port not from the range (under 7000)
2. Remove t.Parallel() from this test

[aLekSer]

In order to try to reproduce scenario, mentioned in comment above, you can do (change ./examples/gameserver.yaml name to generateName):

kubectl apply -f ./examples/simple-udp/fleet.yaml
kubectl get gs

Copy port and change ./examples/gameserver.yaml HostPort to one of HostPort which are used.

$ kubectl get gs | grep ${PORT} | wc -l
1
$ for i in {0..4} ; do kubectl apply -f ./examples/gameserver.yaml ; done
$ kubectl delete fleet simple-udp

Wait for port being deallocated.

$ kubectl get gs | grep ${PORT} | wc -l
3
$ kubectl apply -f ./examples/gameserver.yaml 

Last GameServer could now be in a Ready state, but it is also Unhealthy. But usually pod is started for previous to last GS (it stays in Unhealthy state but pod is running now) and last one N+1 become Unhealthy, but in theory second to last and last could took that freed out port from simple-udp with similar probability.