Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flaky: TestGameServerReserve #1412

Conversation

markmandel
Copy link
Member

The idea behind this solution is that as you get down into the WaitFor() commands, there is a goroutine that is run to implement polling.

Normally, if there is a delay due to goroutine scheduling, this is fine, as we are waiting for something to happen, and if the test comes in after the thing happened - then that is fine.

In this instance, we definitely need to check right away, so I implemented a special case of using a for loop instead, so that the test can't be non-determinately delayed.

The idea behind this solution is that as you get down into the WaitFor()
commands, there is a goroutine that is run to implement polling.

Normally, if there is a delay due to goroutine scheduling, this is fine,
as we are waiting for something to happen, and if the test comes in
after the thing happened - then that is fine.

In this instance, we definitely need to check right away, so I
implemented a special case of using a for loop instead, so that the test
can't be non-determinately delayed.
@markmandel markmandel added kind/bug These are bugs. area/tests Unit tests, e2e tests, anything to make sure things don't break labels Mar 13, 2020
@markmandel markmandel added this to the 1.5.0 milestone Mar 13, 2020
@google-oss-robot
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: markmandel

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@agones-bot
Copy link
Collaborator

Build Succeeded 👏

Build Id: 3575b80c-e2c3-4941-bf80-b296b5abb174

The following development artifacts have been built, and will exist for the next 30 days:

A preview of the website (the last 30 builds are retained):

To install this version:

  • git fetch https://github.com/GoogleCloudPlatform/agones.git pull/1412/head:pr_1412 && git checkout pr_1412
  • helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.5.0-07c43a1

@agones-bot
Copy link
Collaborator

Build Succeeded 👏

Build Id: a67a5001-cda8-4241-961a-0536234f0552

The following development artifacts have been built, and will exist for the next 30 days:

A preview of the website (the last 30 builds are retained):

To install this version:

  • git fetch https://github.com/GoogleCloudPlatform/agones.git pull/1412/head:pr_1412 && git checkout pr_1412
  • helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.5.0-8ec7dc2

@agones-bot
Copy link
Collaborator

Build Succeeded 👏

Build Id: ab62a331-9594-468d-8e93-5e4b0c504f0e

The following development artifacts have been built, and will exist for the next 30 days:

A preview of the website (the last 30 builds are retained):

To install this version:

  • git fetch https://github.com/GoogleCloudPlatform/agones.git pull/1412/head:pr_1412 && git checkout pr_1412
  • helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.5.0-7645b9f

@agones-bot
Copy link
Collaborator

Build Succeeded 👏

Build Id: 5afce8ac-4dbc-4f73-bf0a-c5d41469fbf8

The following development artifacts have been built, and will exist for the next 30 days:

A preview of the website (the last 30 builds are retained):

To install this version:

  • git fetch https://github.com/GoogleCloudPlatform/agones.git pull/1412/head:pr_1412 && git checkout pr_1412
  • helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.5.0-7645b9f

@agones-bot
Copy link
Collaborator

Build Failed 😱

Build Id: c4c50efe-3436-42d6-b367-1005e41bca6c

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

@markmandel
Copy link
Member Author

Different flake - TestControllerRunCacheSync in the unit tests.

@agones-bot
Copy link
Collaborator

Build Succeeded 👏

Build Id: 6015d478-d0f6-4024-bc25-27bfd6355ad3

The following development artifacts have been built, and will exist for the next 30 days:

A preview of the website (the last 30 builds are retained):

To install this version:

  • git fetch https://github.com/GoogleCloudPlatform/agones.git pull/1412/head:pr_1412 && git checkout pr_1412
  • helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.5.0-7645b9f

@agones-bot
Copy link
Collaborator

Build Failed 😱

Build Id: e017ae58-7a57-4f02-a4a4-62eb6776bb91

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

@markmandel
Copy link
Member Author

Failure on different flake - TestUnhealthyGameServersWithoutFreePorts

@agones-bot
Copy link
Collaborator

Build Failed 😱

Build Id: be902966-192a-4df1-a6b0-e52791e0910e

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

@markmandel
Copy link
Member Author

I thought I had solved this, but nope - check this out.

time="2020-03-14 21:21:08.886" level=info msg="sending RESERVE command"
time="2020-03-14 21:21:08.945" level=info msg="Received response"
time="2020-03-14 21:21:08.945" level=info msg="checking for reserved state"
time="2020-03-14 21:21:19.310" level=info msg="Waiting for reserved state" currentState=Ready gs=udp-serverkhnmx
time="2020-03-14 21:21:20.311" level=info msg="checking for reserved state"
time="2020-03-14 21:21:30.467" level=info msg="Waiting for reserved state" currentState=Ready gs=udp-serverkhnmx
time="2020-03-14 21:21:31.467" level=info msg="checking for reserved state"
time="2020-03-14 21:21:40.791" level=info msg="Waiting for reserved state" currentState=Ready gs=udp-serverkhnmx
time="2020-03-14 21:21:41.791" level=info msg="checking for reserved state"
time="2020-03-14 21:21:49.454" level=info msg="Waiting for reserved state" currentState=Ready gs=udp-serverkhnmx
time="2020-03-14 21:21:50.454" level=info msg="checking for reserved state"
time="2020-03-14 21:21:56.697" level=info msg="Waiting for reserved state" currentState=Ready gs=udp-serverkhnmx
time="2020-03-14 21:21:57.697" level=info msg="checking for reserved state"
time="2020-03-14 21:22:03.070" level=info msg="Waiting for reserved state" currentState=Ready gs=udp-serverkhnmx
time="2020-03-14 21:22:04.070" level=info msg="checking for reserved state"
time="2020-03-14 21:22:09.285" level=info msg="Waiting for reserved state" currentState=Ready gs=udp-serverkhnmx
time="2020-03-14 21:22:10.285" level=info msg="checking for reserved state"
time="2020-03-14 21:22:15.262" level=info msg="Waiting for reserved state" currentState=Ready gs=udp-serverkhnmx
time="2020-03-14 21:22:16.263" level=info msg="checking for reserved state"
time="2020-03-14 21:22:21.081" level=info msg="Waiting for reserved state" currentState=Ready gs=udp-serverkhnmx
time="2020-03-14 21:22:22.081" level=info msg="checking for reserved state"
time="2020-03-14 21:22:26.879" level=info msg="Waiting for reserved state" currentState=Ready gs=udp-serverkhnmx
time="2020-03-14 21:22:27.880" level=info msg="checking for reserved state"
time="2020-03-14 21:22:32.067" level=info msg="Waiting for reserved state" currentState=Ready gs=udp-serverkhnmx
time="2020-03-14 21:22:33.067" level=info msg="checking for reserved state"
time="2020-03-14 21:22:36.666" level=info msg="Waiting for reserved state" currentState=Ready gs=udp-serverkhnmx
time="2020-03-14 21:22:37.667" level=info msg="checking for reserved state"
time="2020-03-14 21:22:40.667" level=info msg="Waiting for reserved state" currentState=Ready gs=udp-serverkhnmx
time="2020-03-14 21:22:41.667" level=info msg="checking for reserved state"
time="2020-03-14 21:22:44.320" level=info msg="Waiting for reserved state" currentState=Ready gs=udp-serverkhnmx
time="2020-03-14 21:22:45.320" level=info msg="checking for reserved state"
time="2020-03-14 21:22:46.873" level=info msg="Waiting for reserved state" currentState=Ready gs=udp-serverkhnmx
time="2020-03-14 21:22:49.470" level=info msg="Waiting for states to match" awaitingState=Ready currentState=Ready gs=udp-serverkhnmx
time="2020-03-14 21:22:51.163" level=info msg="gs event details" count=1 first-time="2020-03-14 21:20:35 +0000 UTC" last-time="2020-03-14 21:20:35 +0000 UTC" message="Port allocated" name=udp-serverkhnmx.15fc4874d2bc0c31 reason=PortAllocation
time="2020-03-14 21:22:51.163" level=info msg="gs event details" count=1 first-time="2020-03-14 21:20:35 +0000 UTC" last-time="2020-03-14 21:20:35 +0000 UTC" message="Pod udp-serverkhnmx created" name=udp-serverkhnmx.15fc4874df836425 reason=Creating
time="2020-03-14 21:22:51.163" level=info msg="gs event details" count=1 first-time="2020-03-14 21:20:35 +0000 UTC" last-time="2020-03-14 21:20:35 +0000 UTC" message="Address and port populated" name=udp-serverkhnmx.15fc4874e4b9e1bc reason=Scheduled
time="2020-03-14 21:22:51.163" level=info msg="gs event details" count=2 first-time="2020-03-14 21:20:47 +0000 UTC" last-time="2020-03-14 21:21:18 +0000 UTC" message="SDK state change" name=udp-serverkhnmx.15fc487790badfb2 reason=RequestReady
time="2020-03-14 21:22:51.163" level=info msg="gs event details" count=2 first-time="2020-03-14 21:21:03 +0000 UTC" last-time="2020-03-14 21:21:19 +0000 UTC" message="SDK.Ready() complete" name=udp-serverkhnmx.15fc487b3c2777d4 reason=Ready
time="2020-03-14 21:22:51.164" level=info msg="gs event details" count=1 first-time="2020-03-14 21:21:08 +0000 UTC" last-time="2020-03-14 21:21:08 +0000 UTC" message="SDK state change, for 10s" name=udp-serverkhnmx.15fc487c9d04edfe reason=Reserved
time="2020-03-14 21:22:51.389" level=info msg="Pod Status" status="{Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2020-03-14 21:20:35 +0000 UTC  } {Ready True 0001-01-01 00:00:00 +0000 UTC 2020-03-14 21:21:01 +0000 UTC  } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2020-03-14 21:21:01 +0000 UTC  } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2020-03-14 21:20:35 +0000 UTC  }]    10.138.15.238 10.40.2.21 2020-03-14 21:20:35 +0000 UTC [] [{agones-gameserver-sidecar {nil &ContainerStateRunning{StartedAt:2020-03-14 21:20:44 +0000 UTC,} nil} {nil nil nil} true 0 gcr.io/agones-images/agones-sdk:1.5.0-7645b9f docker-pullable://gcr.io/agones-images/agones-sdk@sha256:2ca400f2ed93a6835024e44b16a546ac0b5e79fb02e581d5f3969fc9b4ce44af docker://4af28c99104718a8ffcf787281633c306d84769c9236560a92f24009b2d257f2} {udp-server {nil &ContainerStateRunning{StartedAt:2020-03-14 21:20:40 +0000 UTC,} nil} {nil nil nil} true 0 gcr.io/agones-images/udp-server:0.18 docker-pullable://gcr.io/agones-images/udp-server@sha256:fa64d524c817cbb8f23c59214c51b039148358bd50623ec14843e5fc83284a25 docker://a487c34962e09db5c7d417374af33859cbe33a121c5a46323ae53a2c0cf0f5e1}] Burstable}"
time="2020-03-14 21:22:53.126" level=info msg="gs pod details" count=1 first-time="2020-03-14 21:20:35 +0000 UTC" last-time="2020-03-14 21:20:35 +0000 U