Initial version of SDK conformance Testing framework for Golang, NodeJS and others

[aLekSer]

PoC for SDK conformance testing.
Proposal Document could be found here.
Test engine performs two tasks: code generation for Golang, Node SDKs and verification that SDK sends and sidecar receives all GRPC messages as intended.
YAML file with test definition in Formal language -> translates and concatenates from excerpts in SDK target language.
Code generator use code templates from sdk_client use test Case
scenarios from harness folder and produce go file which would run the
scenario.
make build-conformance-tests - build docker image and binaries of sdk-server (sidecar) and test_engine
make run-conformance-tests - build binaries and perform verify step (also run-conformance-jstests):

./sdk --verify=true --sdk=nodejs - runs sdk-server and binaries of target SDK client generated before.
Added --timeout for sdk-server (Sidecar) local run.

For #672.

[agones-bot]

Build Failed 😱

Build Id: 294d22d4-ce75-4e55-8fc5-b4e470fa7ac7

• Cloud Build view
• Cloud Build log download

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

[agones-bot]

Build Succeeded 👏

Build Id: 7e9ae2c5-5b7f-45e3-b160-4fa9df2c081f

The following development artifacts have been built, and will exist for the next 30 days:

• image: gcr.io/agones-images/agones-controller:0.11.0-cc31b96
• image: gcr.io/agones-images/agones-sdk:0.11.0-cc31b96
• image: gcr.io/agones-images/agones-ping:0.11.0-cc31b96
• Linux C++ SDK (build): agonessdk-0.11.0-cc31b96-linux-arch_64.tar.gz
• SDK Server: agonessdk-server-0.11.0-cc31b96.zip

A preview of the website (the last 30 builds are retained):

• https://cc31b96-dot-preview-dot-agones-images.appspot.com/

To install this version:

• git fetch https://github.com/GoogleCloudPlatform/agones.git pull/845/head:pr_845 && git checkout pr_845
• helm install install/helm/agones --namespace agones-system --name agones --set agones.image.tag=0.11.0-cc31b96

[agones-bot]

Build Succeeded 👏

Build Id: fd299554-9213-41f3-a582-a84865b234fa

The following development artifacts have been built, and will exist for the next 30 days:

• image: gcr.io/agones-images/agones-controller:0.11.0-73ee02a
• image: gcr.io/agones-images/agones-sdk:0.11.0-73ee02a
• image: gcr.io/agones-images/agones-ping:0.11.0-73ee02a
• Linux C++ SDK (build): agonessdk-0.11.0-73ee02a-linux-arch_64.tar.gz
• SDK Server: agonessdk-server-0.11.0-73ee02a.zip

A preview of the website (the last 30 builds are retained):

• https://73ee02a-dot-preview-dot-agones-images.appspot.com/

To install this version:

• git fetch https://github.com/GoogleCloudPlatform/agones.git pull/845/head:pr_845 && git checkout pr_845
• helm install install/helm/agones --namespace agones-system --name agones --set agones.image.tag=0.11.0-73ee02a

[agones-bot]

Build Succeeded 👏

Build Id: 6889bd00-9055-4700-9f28-39acbe771833

The following development artifacts have been built, and will exist for the next 30 days:

• image: gcr.io/agones-images/agones-controller:0.11.0-6c46821
• image: gcr.io/agones-images/agones-sdk:0.11.0-6c46821
• image: gcr.io/agones-images/agones-ping:0.11.0-6c46821
• Linux C++ SDK (build): agonessdk-0.11.0-6c46821-linux-arch_64.tar.gz
• SDK Server: agonessdk-server-0.11.0-6c46821.zip

A preview of the website (the last 30 builds are retained):

• https://6c46821-dot-preview-dot-agones-images.appspot.com/

To install this version:

• git fetch https://github.com/GoogleCloudPlatform/agones.git pull/845/head:pr_845 && git checkout pr_845
• helm install install/helm/agones --namespace agones-system --name agones --set agones.image.tag=0.11.0-6c46821

[markmandel]

So I'm digging through this, and while it seems cool - I'm wondering if it's over-engineered?

In the model in my head, there would be a Docker container, with a builder pattern for a language that gets built and run on each test for each SDK.

Alongside it we could run the standard local sdk server, with some variation of:
docker run --rm gcr.io/agones-images/sdk-server:0.11.0 --test-mode=true, --test-list=init,get,watch,allocate

The test mode expects those tests to pass (they have documented requirements) within,1 minute (or whatever time period - 30s may be enough)

Then we don't need code generation, or yaml files, or a go test hardness, or most of the complicated pieces.

This also makes it very easy for anyone building their own custom SDK to test against our testbed, without having to dig into all these harness pieces.

WDYT?

[aLekSer]

Thanks for the review, Mark.
Yes, my solution seems to be more complicated than it is needed. At least I understood SDK a lot better.
One thing which we could not control when we just have the acceptance test on sdk-server side is the parsing the results on SDK client side and need to change more on sidecar side. But for the burden on SDK developer seems to be the same in both options of realisation.
I am going to develop proposed solution by you. Let's leave this PR for a while till I prepare a new one.

[aLekSer]

This one was superseded by #848 as it would be easier to maintain.

[aLekSer]

Remove this PR as we switched to another version without code generation.