Switch to parrallel execution of SDK commands #742

aLekSer · 2019-04-24T09:50:13Z

Use subshell execution of the run-all-sdk-command make target.

agones-bot · 2019-04-24T10:06:05Z

Build Succeeded 👏

Build Id: 7ad13ad2-04bc-4cd2-ab8c-ea3826632791

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:0.10.0-b74206b
image: gcr.io/agones-images/agones-sdk:0.10.0-b74206b
image: gcr.io/agones-images/agones-ping:0.10.0-b74206b
Linux C++ SDK (build): agonessdk-0.10.0-b74206b-linux-arch_64.tar.gz
SDK Server: agonessdk-server-0.10.0-b74206b.zip

A preview of the website (the last 30 builds are retained):

https://b74206b-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/GoogleCloudPlatform/agones.git pull/742/head:pr_742 && git checkout pr_742
helm install install/helm/agones --namespace agones-system --name agones --set agones.image.tag=0.10.0-b74206b

aLekSer · 2019-04-24T11:09:12Z

I have added wait right after the loop, if one process fail we would get the error and a message.
Tested with make test-sdks it spawns two docker containers for go and node.

And tested errors scenario for instance if we add exit 1 to test.sh script. That would result in:

17 specs, 0 failures
Finished in 0.087 seconds
Failed to run SDK command
make: *** [run-all-sdk-command] Error 255

Note that we only have one build.sh script (currently for cpp only). So running make build-sdks would run only one container.

agones-bot · 2019-04-24T11:09:32Z

Build Succeeded 👏

Build Id: 7e353e99-3d43-45a1-bccf-099a02d6ae78

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:0.10.0-6711531
image: gcr.io/agones-images/agones-sdk:0.10.0-6711531
image: gcr.io/agones-images/agones-ping:0.10.0-6711531
Linux C++ SDK (build): agonessdk-0.10.0-6711531-linux-arch_64.tar.gz
SDK Server: agonessdk-server-0.10.0-6711531.zip

A preview of the website (the last 30 builds are retained):

https://6711531-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/GoogleCloudPlatform/agones.git pull/742/head:pr_742 && git checkout pr_742
helm install install/helm/agones --namespace agones-system --name agones --set agones.image.tag=0.10.0-6711531

agones-bot · 2019-04-24T11:13:06Z

Build Succeeded 👏

Build Id: 251a76ce-c16e-41a2-b6f0-d2e33abac867

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:0.10.0-2232de4
image: gcr.io/agones-images/agones-sdk:0.10.0-2232de4
image: gcr.io/agones-images/agones-ping:0.10.0-2232de4
Linux C++ SDK (build): agonessdk-0.10.0-2232de4-linux-arch_64.tar.gz
SDK Server: agonessdk-server-0.10.0-2232de4.zip

A preview of the website (the last 30 builds are retained):

https://2232de4-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/GoogleCloudPlatform/agones.git pull/742/head:pr_742 && git checkout pr_742
helm install install/helm/agones --namespace agones-system --name agones --set agones.image.tag=0.10.0-2232de4

agones-bot · 2019-04-24T11:16:19Z

Build Succeeded 👏

Build Id: 41249e09-c5c0-498a-988c-e0461fdbf362

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:0.10.0-f5c26d1
image: gcr.io/agones-images/agones-sdk:0.10.0-f5c26d1
image: gcr.io/agones-images/agones-ping:0.10.0-f5c26d1
Linux C++ SDK (build): agonessdk-0.10.0-f5c26d1-linux-arch_64.tar.gz
SDK Server: agonessdk-server-0.10.0-f5c26d1.zip

A preview of the website (the last 30 builds are retained):

https://f5c26d1-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/GoogleCloudPlatform/agones.git pull/742/head:pr_742 && git checkout pr_742
helm install install/helm/agones --namespace agones-system --name agones --set agones.image.tag=0.10.0-f5c26d1

jkowalski · 2019-04-24T14:25:48Z

build/includes/sdk.mk

@@ -55,17 +55,22 @@ gen-all-sdk-grpc: run-all-sdk-command
 gen-sdk-grpc: COMMAND := gen
 gen-sdk-grpc: run-sdk-command

-# Runs a command on all supported languages, use COMMAND variable to select which command.
+# Runs a command on all supported languages in parallel, use COMMAND variable to select which command.


how about using native parallelism built into make:

run-all-sdk-command: run-sdk-command-go run-sdk-command-rust run-sdk-command-cpp run-sdk-command-cpp: $(MAKE) run-sdk-command SDK_FOLDER=cpp run-sdk-command-rust: $(MAKE) run-sdk-command SDK_FOLDER=rust run-sdk-command-go: $(MAKE) run-sdk-command SDK_FOLDER=go

Then the user can do make -j4 run-all-sdk-command COMMAND=gen

ooh that's a really interesting idea. I wonder if we could also come up with a pattern rule so we don't have to define each rule individually?

That native way looks more natural and simple, updating this Pull Request, removing initial and next waiting for loop:

for job in `jobs -p` ; \ do \ wait $$job || let "FAIL+=1"; \ done ; \

@jkowalski
The only thing that we don't have build.sh in each directory, that's why we have such check that file exists:
if [ "$${d%?}" != "tool" ] && [ -f $$d/$(COMMAND).sh ];

markmandel · 2019-04-24T17:49:13Z

/cc @Kuqd any thoughts?

agones-bot · 2019-04-25T13:04:11Z

Build Succeeded 👏

Build Id: d1ff4a7e-44c2-451d-a5b9-50004a0f2508

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:0.10.0-68b65f8
image: gcr.io/agones-images/agones-sdk:0.10.0-68b65f8
image: gcr.io/agones-images/agones-ping:0.10.0-68b65f8
Linux C++ SDK (build): agonessdk-0.10.0-68b65f8-linux-arch_64.tar.gz
SDK Server: agonessdk-server-0.10.0-68b65f8.zip

A preview of the website (the last 30 builds are retained):

https://68b65f8-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/GoogleCloudPlatform/agones.git pull/742/head:pr_742 && git checkout pr_742
helm install install/helm/agones --namespace agones-system --name agones --set agones.image.tag=0.10.0-68b65f8

agones-bot · 2019-04-25T13:17:05Z

Build Succeeded 👏

Build Id: 4c207497-f260-46fc-983c-fdf77a407093

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:0.10.0-9cab1b0
image: gcr.io/agones-images/agones-sdk:0.10.0-9cab1b0
image: gcr.io/agones-images/agones-ping:0.10.0-9cab1b0
Linux C++ SDK (build): agonessdk-0.10.0-9cab1b0-linux-arch_64.tar.gz
SDK Server: agonessdk-server-0.10.0-9cab1b0.zip

A preview of the website (the last 30 builds are retained):

https://9cab1b0-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/GoogleCloudPlatform/agones.git pull/742/head:pr_742 && git checkout pr_742
helm install install/helm/agones --namespace agones-system --name agones --set agones.image.tag=0.10.0-9cab1b0

aLekSer · 2019-04-25T13:20:24Z

Updated cloudbuild-yaml also. Now if I run:
make -j 5 test
I would see 5 parallel running images as expected (docker ps --format "table {{.Image}}\t{{.Command}}"):

IMAGE                              COMMAND                                                                                                                  
agones-build-sdk-go:c168fcc53f     "/root/entrypoint.sh…"                                                                                             
agones-build-sdk-node:c168fcc53f   "/root/entrypoint.sh…"                                                                                             
agones-build:c168fcc53f            "bash -c /go/src/ago…"                                                                                             
agones-build:c168fcc53f            "go test -mod=vendor…"                                                                                             
agones-build:c168fcc53f            "bash -c /go/src/ago…"

Combine multiple targets into one `run-all-sdk-command` make target. Now we can run make -j 2 test-sdks.

agones-bot · 2019-04-29T12:14:24Z

Build Succeeded 👏

Build Id: 94f5a06c-a30c-44c2-888e-8575ad434dbe

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:0.10.0-2e1a2cb
image: gcr.io/agones-images/agones-sdk:0.10.0-2e1a2cb
image: gcr.io/agones-images/agones-ping:0.10.0-2e1a2cb
Linux C++ SDK (build): agonessdk-0.10.0-2e1a2cb-linux-arch_64.tar.gz
SDK Server: agonessdk-server-0.10.0-2e1a2cb.zip

A preview of the website (the last 30 builds are retained):

https://2e1a2cb-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/GoogleCloudPlatform/agones.git pull/742/head:pr_742 && git checkout pr_742
helm install install/helm/agones --namespace agones-system --name agones --set agones.image.tag=0.10.0-2e1a2cb

markmandel

👍

aLekSer force-pushed the run-sdk-in-parallel branch 3 times, most recently from 2232de4 to f5c26d1 Compare April 24, 2019 11:01

aLekSer marked this pull request as ready for review April 24, 2019 11:04

jkowalski reviewed Apr 24, 2019

View reviewed changes

aLekSer force-pushed the run-sdk-in-parallel branch 2 times, most recently from 68b65f8 to 9cab1b0 Compare April 25, 2019 13:02

markmandel added area/build-tools Development tooling. I.e. pretty much everything in the `build` directory. kind/cleanup Refactoring code, fixing up documentation, etc labels Apr 27, 2019

Switch to parrallel execution of SDK commands

2e1a2cb

Combine multiple targets into one `run-all-sdk-command` make target. Now we can run make -j 2 test-sdks.

aLekSer force-pushed the run-sdk-in-parallel branch from 9cab1b0 to 2e1a2cb Compare April 29, 2019 11:59

markmandel approved these changes Apr 30, 2019

View reviewed changes

markmandel merged commit 1cc5339 into googleforgames:master Apr 30, 2019

markmandel added this to the 0.10.0 milestone May 7, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Switch to parrallel execution of SDK commands #742

Switch to parrallel execution of SDK commands #742

aLekSer commented Apr 24, 2019

agones-bot commented Apr 24, 2019

aLekSer commented Apr 24, 2019 •

edited

Loading

agones-bot commented Apr 24, 2019

agones-bot commented Apr 24, 2019

agones-bot commented Apr 24, 2019

jkowalski Apr 24, 2019

markmandel Apr 24, 2019

aLekSer Apr 25, 2019

aLekSer Apr 25, 2019

markmandel commented Apr 24, 2019

agones-bot commented Apr 25, 2019

agones-bot commented Apr 25, 2019

aLekSer commented Apr 25, 2019

agones-bot commented Apr 29, 2019

markmandel left a comment

Switch to parrallel execution of SDK commands #742

Switch to parrallel execution of SDK commands #742

Conversation

aLekSer commented Apr 24, 2019

agones-bot commented Apr 24, 2019

aLekSer commented Apr 24, 2019 • edited Loading

agones-bot commented Apr 24, 2019

agones-bot commented Apr 24, 2019

agones-bot commented Apr 24, 2019

jkowalski Apr 24, 2019

Choose a reason for hiding this comment

markmandel Apr 24, 2019

Choose a reason for hiding this comment

aLekSer Apr 25, 2019

Choose a reason for hiding this comment

aLekSer Apr 25, 2019

Choose a reason for hiding this comment

markmandel commented Apr 24, 2019

agones-bot commented Apr 25, 2019

agones-bot commented Apr 25, 2019

aLekSer commented Apr 25, 2019

agones-bot commented Apr 29, 2019

markmandel left a comment

Choose a reason for hiding this comment

aLekSer commented Apr 24, 2019 •

edited

Loading