Add gRPC load test for allocation service #1867

ilkercelikyilmaz · 2020-10-24T02:14:44Z

What type of PR is this?
/kind documentation

What this PR does / Why we need it:
This adds a sample grpc load test client for the allocation service.

Which issue(s) this PR fixes:
Provides allocation load test client and instructions

Special notes for your reviewer:

agones-bot · 2020-10-24T02:23:33Z

Build Failed 😱

Build Id: 06927514-2130-40bf-a55c-9e016d3065c7

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

test/load/allocation/grpc/README.md

test/load/allocation/grpc/allocationload.go

test/load/allocation/grpc/README.md

agones-bot · 2020-10-25T23:43:48Z

Build Succeeded 👏

Build Id: 1d20b468-10e5-4162-a018-b7506d92511f

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:1.10.0-f224beb
image: gcr.io/agones-images/agones-sdk:1.10.0-f224beb
image: gcr.io/agones-images/agones-ping:1.10.0-f224beb
Linux C++ SDK (build): agonessdk-1.10.0-f224beb-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.10.0-f224beb.zip

A preview of the website (the last 30 builds are retained):

https://f224beb-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/1867/head:pr_1867 && git checkout pr_1867
helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.10.0-f224beb

agones-bot · 2020-10-25T23:49:00Z

Build Succeeded 👏

Build Id: 77dc3ef7-2f28-4be8-a24c-4c9020cccd9f

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:1.10.0-ee41749
image: gcr.io/agones-images/agones-sdk:1.10.0-ee41749
image: gcr.io/agones-images/agones-ping:1.10.0-ee41749
Linux C++ SDK (build): agonessdk-1.10.0-ee41749-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.10.0-ee41749.zip

A preview of the website (the last 30 builds are retained):

https://ee41749-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/1867/head:pr_1867 && git checkout pr_1867
helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.10.0-ee41749

pooneh-m · 2020-10-26T17:00:06Z

test/load/allocation/grpc/allocationload.go

+		panic(err)
+	}
+
+	grpcClient := pb.NewAllocationServiceClient(conn)


Should the gRPC client be created within the go routing for per client?
That would be a closer per client simulation.

I tried it inside the loop and performance was same. See the update

Thanks for the change. You should also add dial to the per client go routine.

That reduces the throughput. We can consider different test such as 5 MM client (e.g one per continent) with concurrent allocations using the same client

I made the change. With 40 clients, we get few more errors (still reasonable). To allocate 5000 GSs with 40 clients, it takes ~70 secs. To increase throughput, we probably need more allocator service replicas

I think if you only use one connection, that connection goes to the same pod, therefore you are not testing multiple clients/pods.

I think if you only use one connection, that connection goes to the same pod, therefore you are not testing multiple clients/pods.

This is an interesting question actually. But a couple of thoughts:

I think you are right @pooneh-m - and we want to test across multiple pods. But I'm not sure if that's the requirement for this PR.

Do we expect someone is going to have 50 matchmakers? Seems unlikely - so having a single client may actually make sense, since that's what most architectures will have (or at least, a very low number)

(Separate to this PR) Should we look at how we can make it easier to better implement gRPC load balancing? Some resources:

https://grpc.io/blog/grpc-load-balancing/

https://blog.nobugware.com/post/2019/kubernetes_mesh_network_load_balancing_grpc_services/

https://kubernetes.io/blog/2018/11/07/grpc-load-balancing-on-kubernetes-without-tears/

https://itnext.io/proxyless-grpc-load-balancing-in-kubernetes-ca1a4797b742

https://cloud.google.com/solutions/exposing-grpc-services-on-gke-using-envoy-proxy

If so, this sounds like something we should track in a separate issue for discussion.

I think it is a good discussion.

The current load test client, creates one connection per client so it should leverage all the pods. The performance is more or less so keeping it as is.

Started an issue: #1872

test/load/allocation/grpc/README.md

test/load/allocation/grpc/runAllocation.sh

agones-bot · 2020-10-26T23:22:21Z

Build Failed 😱

Build Id: c33e9c49-c98f-4e61-9ad0-57ec209c9a40

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

markmandel · 2020-10-26T23:31:08Z

Linter fail:

make[1]: Leaving directory '/workspace/build'
docker run -t -e "TERM=xterm-256color" -e "GO111MODULE=on" --rm -v /workspace/build//.config/gcloud:/root/.config/gcloud -v ~/.kube/:/root/.kube -v ~/.config/helm:/root/.config/helm -v ~/.cache/helm:/root/.cache/helm -v /workspace:/go/src/agones.dev/agones -v /workspace/build//.gomod:/go/pkg/mod -v /workspace/build//.gocache:/root/.cache/go-build -w /go/src/agones.dev/agones  agones-build:724013197f bash -c \
	"golangci-lint run ./examples/... && golangci-lint run --timeout 15m ./..."
test/load/allocation/grpc/allocationload.go:98:12: Error return value of `conn.Close` is not checked (errcheck)
	conn.Close()
	          ^
Makefile:331: recipe for target 'lint' failed
make: *** [lint] Error 1

agones-bot · 2020-10-26T23:45:58Z

Build Failed 😱

Build Id: 85e728d4-b108-4bfa-b389-c46bb3312047

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

test/load/allocation/grpc/runAllocation.sh

agones-bot · 2020-10-27T00:06:24Z

Build Succeeded 👏

Build Id: d6fce878-c83e-45dd-954a-ab0bc7a90b7a

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:1.10.0-500d8ef
image: gcr.io/agones-images/agones-sdk:1.10.0-500d8ef
image: gcr.io/agones-images/agones-ping:1.10.0-500d8ef
Linux C++ SDK (build): agonessdk-1.10.0-500d8ef-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.10.0-500d8ef.zip

A preview of the website (the last 30 builds are retained):

https://500d8ef-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/1867/head:pr_1867 && git checkout pr_1867
helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.10.0-500d8ef

agones-bot · 2020-10-27T02:49:45Z

Build Succeeded 👏

Build Id: 1a15fae9-f3cb-4e40-b4fa-19a820f2ceea

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:1.10.0-711a1ca
image: gcr.io/agones-images/agones-sdk:1.10.0-711a1ca
image: gcr.io/agones-images/agones-ping:1.10.0-711a1ca
Linux C++ SDK (build): agonessdk-1.10.0-711a1ca-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.10.0-711a1ca.zip

A preview of the website (the last 30 builds are retained):

https://711a1ca-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/1867/head:pr_1867 && git checkout pr_1867
helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.10.0-711a1ca

agones-bot · 2020-10-27T03:25:44Z

Build Succeeded 👏

Build Id: c7888bc3-3fbe-4560-85de-08aac01d0618

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:1.10.0-3439241
image: gcr.io/agones-images/agones-sdk:1.10.0-3439241
image: gcr.io/agones-images/agones-ping:1.10.0-3439241
Linux C++ SDK (build): agonessdk-1.10.0-3439241-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.10.0-3439241.zip

A preview of the website (the last 30 builds are retained):

https://3439241-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/1867/head:pr_1867 && git checkout pr_1867
helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.10.0-3439241

test/load/allocation/grpc/allocationload.go

test/load/allocation/grpc/fleet.yaml

agones-bot · 2020-10-27T21:52:30Z

Build Succeeded 👏

Build Id: 8cdcc050-5a6b-470a-914d-716eadf3bfa9

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:1.10.0-9122913
image: gcr.io/agones-images/agones-sdk:1.10.0-9122913
image: gcr.io/agones-images/agones-ping:1.10.0-9122913
Linux C++ SDK (build): agonessdk-1.10.0-9122913-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.10.0-9122913.zip

A preview of the website (the last 30 builds are retained):

https://9122913-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/1867/head:pr_1867 && git checkout pr_1867
helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.10.0-9122913

agones-bot · 2020-10-27T22:48:09Z

Build Succeeded 👏

Build Id: 2204b1ae-acc5-4174-9bf0-ce634e95b0e5

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:1.10.0-ae480c2
image: gcr.io/agones-images/agones-sdk:1.10.0-ae480c2
image: gcr.io/agones-images/agones-ping:1.10.0-ae480c2
Linux C++ SDK (build): agonessdk-1.10.0-ae480c2-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.10.0-ae480c2.zip

A preview of the website (the last 30 builds are retained):

https://ae480c2-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/1867/head:pr_1867 && git checkout pr_1867
helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.10.0-ae480c2

markmandel · 2020-10-27T22:49:28Z

Other than that one small nit in the fleet.yaml, this is good to go for me!

agones-bot · 2020-10-28T03:19:35Z

Build Succeeded 👏

Build Id: fef813d5-673d-45ff-9964-b0ad00a2a6c7

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:1.10.0-f69c05a
image: gcr.io/agones-images/agones-sdk:1.10.0-f69c05a
image: gcr.io/agones-images/agones-ping:1.10.0-f69c05a
Linux C++ SDK (build): agonessdk-1.10.0-f69c05a-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.10.0-f69c05a.zip

A preview of the website (the last 30 builds are retained):

https://f69c05a-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/1867/head:pr_1867 && git checkout pr_1867
helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.10.0-f69c05a

agones-bot · 2020-10-28T03:24:55Z

Build Succeeded 👏

Build Id: 78da6162-e170-4345-bfae-9d2294846234

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:1.10.0-71e3e32
image: gcr.io/agones-images/agones-sdk:1.10.0-71e3e32
image: gcr.io/agones-images/agones-ping:1.10.0-71e3e32
Linux C++ SDK (build): agonessdk-1.10.0-71e3e32-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.10.0-71e3e32.zip

A preview of the website (the last 30 builds are retained):

https://71e3e32-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/1867/head:pr_1867 && git checkout pr_1867
helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.10.0-71e3e32

markmandel · 2020-10-28T03:31:56Z

LGTM! Anything left outstanding for you @pooneh-m or @roberthbailey ?

pooneh-m

LGTM. Thanks for adding the load test @ilkercelikyilmaz! Very useful.

google-oss-robot · 2020-10-28T17:31:27Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ilkercelikyilmaz, pooneh-m

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [pooneh-m]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

google-oss-robot · 2020-10-28T17:36:47Z

New changes are detected. LGTM label has been removed.

agones-bot · 2020-10-28T17:59:26Z

Build Succeeded 👏

Build Id: 57474ca2-9588-4533-9de3-58ea42371541

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:1.10.0-6ba7e4a
image: gcr.io/agones-images/agones-sdk:1.10.0-6ba7e4a
image: gcr.io/agones-images/agones-ping:1.10.0-6ba7e4a
Linux C++ SDK (build): agonessdk-1.10.0-6ba7e4a-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.10.0-6ba7e4a.zip

A preview of the website (the last 30 builds are retained):

https://6ba7e4a-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/1867/head:pr_1867 && git checkout pr_1867
helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.10.0-6ba7e4a

google-cla bot added the cla: yes label Oct 24, 2020

google-oss-robot requested review from markmandel and pooneh-m October 24, 2020 02:14

google-oss-robot added the size/L label Oct 24, 2020

roberthbailey reviewed Oct 24, 2020

View reviewed changes

pooneh-m reviewed Oct 26, 2020

View reviewed changes

markmandel reviewed Oct 26, 2020

View reviewed changes

test/load/allocation/grpc/README.md Outdated Show resolved Hide resolved

markmandel reviewed Oct 26, 2020

View reviewed changes

test/load/allocation/grpc/README.md Outdated Show resolved Hide resolved

markmandel reviewed Oct 26, 2020

View reviewed changes

test/load/allocation/grpc/runAllocation.sh Outdated Show resolved Hide resolved

markmandel reviewed Oct 26, 2020

View reviewed changes

test/load/allocation/grpc/runAllocation.sh Outdated Show resolved Hide resolved

ilkercelikyilmaz added 3 commits October 26, 2020 16:08

Add gRPC load test for allocation service

927c375

Addressed comments

dd9e892

Create gRPClient instance per client

89cab54

ilkercelikyilmaz force-pushed the grpc-loadtest-for-allocation branch from f224beb to 89cab54 Compare October 26, 2020 23:13

ilkercelikyilmaz added 2 commits October 26, 2020 16:37

Addressed Mark's comments

11e0821

Fixed lnt issue

500d8ef

pooneh-m reviewed Oct 27, 2020

View reviewed changes

test/load/allocation/grpc/runAllocation.sh Show resolved Hide resolved

markmandel mentioned this pull request Oct 27, 2020

Built tools: Update install with Allocation certs #1869

Merged

Create the TLS and mTLS cert and key files as part of runAllocation.sh

711a1ca

ilkercelikyilmaz added 2 commits October 26, 2020 20:02

Add grpc.Dial to the for loop to create connection for each client

3439241

Updated the README file related to runallocation.sh file params

9fbdc8c

markmandel reviewed Oct 27, 2020

View reviewed changes

test/load/allocation/grpc/allocationload.go Outdated Show resolved Hide resolved

markmandel reviewed Oct 27, 2020

View reviewed changes

test/load/allocation/grpc/fleet.yaml Outdated Show resolved Hide resolved

Addressed Mark's comments and fixed the bug in fleet.yaml

9122913

ilkercelikyilmaz force-pushed the grpc-loadtest-for-allocation branch from 77e6c68 to 9122913 Compare October 27, 2020 21:25

Add a sample for env variable usage

ae480c2

This was referenced Oct 27, 2020

Allocation Endpoint - Better load balancing for gRPC connection #1872

Closed

Add QPS settings to Allocation endpoints #1863

Merged

ilkercelikyilmaz added 2 commits October 27, 2020 19:55

Removed health from the fleet.yaml

71e3e32

Merge branch 'master' into grpc-loadtest-for-allocation

f69c05a

pooneh-m approved these changes Oct 28, 2020

View reviewed changes

google-oss-robot assigned pooneh-m Oct 28, 2020

google-oss-robot added the lgtm label Oct 28, 2020

google-oss-robot added the approved label Oct 28, 2020

markmandel mentioned this pull request Oct 28, 2020

Upgrade to Kubernetes 1.17 #1824

Closed

34 tasks

Merge branch 'master' into grpc-loadtest-for-allocation

6ba7e4a

google-oss-robot removed the lgtm label Oct 28, 2020

ilkercelikyilmaz merged commit 7bd7139 into googleforgames:master Oct 28, 2020

markmandel added this to the 1.10.0 milestone Nov 3, 2020

markmandel added area/build-tools Development tooling. I.e. pretty much everything in the `build` directory. area/performance Anything to do with Agones being slow, or making it go faster. kind/feature New features for Agones labels Nov 3, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add gRPC load test for allocation service #1867

Add gRPC load test for allocation service #1867

ilkercelikyilmaz commented Oct 24, 2020

agones-bot commented Oct 24, 2020

agones-bot commented Oct 25, 2020

agones-bot commented Oct 25, 2020

pooneh-m Oct 26, 2020

ilkercelikyilmaz Oct 26, 2020

pooneh-m Oct 26, 2020

ilkercelikyilmaz Oct 27, 2020

ilkercelikyilmaz Oct 27, 2020

pooneh-m Oct 27, 2020

markmandel Oct 27, 2020 •

edited

Loading

ilkercelikyilmaz Oct 27, 2020

markmandel Oct 27, 2020

agones-bot commented Oct 26, 2020

markmandel commented Oct 26, 2020

agones-bot commented Oct 26, 2020

agones-bot commented Oct 27, 2020

agones-bot commented Oct 27, 2020

agones-bot commented Oct 27, 2020

agones-bot commented Oct 27, 2020

agones-bot commented Oct 27, 2020

markmandel commented Oct 27, 2020

agones-bot commented Oct 28, 2020

agones-bot commented Oct 28, 2020

markmandel commented Oct 28, 2020

pooneh-m left a comment

google-oss-robot commented Oct 28, 2020

google-oss-robot commented Oct 28, 2020

agones-bot commented Oct 28, 2020

Add gRPC load test for allocation service #1867

Add gRPC load test for allocation service #1867

Conversation

ilkercelikyilmaz commented Oct 24, 2020

agones-bot commented Oct 24, 2020

agones-bot commented Oct 25, 2020

agones-bot commented Oct 25, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

markmandel Oct 27, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

agones-bot commented Oct 26, 2020

markmandel commented Oct 26, 2020

agones-bot commented Oct 26, 2020

agones-bot commented Oct 27, 2020

agones-bot commented Oct 27, 2020

agones-bot commented Oct 27, 2020

agones-bot commented Oct 27, 2020

agones-bot commented Oct 27, 2020

markmandel commented Oct 27, 2020

agones-bot commented Oct 28, 2020

agones-bot commented Oct 28, 2020

markmandel commented Oct 28, 2020

pooneh-m left a comment

Choose a reason for hiding this comment

google-oss-robot commented Oct 28, 2020

google-oss-robot commented Oct 28, 2020

agones-bot commented Oct 28, 2020

markmandel Oct 27, 2020 •

edited

Loading