Migrate from github.com/golang/protobuf to google.golang.org/protobuf

[govargo]

What type of PR is this?

/kind breaking
/kind cleanup
/kind feature

What this PR does / Why we need it:

I found that the github.com/golang/protobuf will be deprecated in the future when I executed make gen-all-sdk-grpc.

+ protoc -I /go/src/agones.dev/agones/proto/googleapis -I /go/src/agones.dev/agones/proto/sdk sdk.proto --grpc-gateway_out=logtostderr=true:pkg/sdk
WARNING: Package "github.com/golang/protobuf/protoc-gen-go/generator" is deprecated.
        A future release of golang/protobuf will delete this package,
        which has long been excluded from the compatibility promise.

+ protoc -I /go/src/agones.dev/agones/proto/googleapis -I /go/src/agones.dev/agones/proto/sdk/alpha alpha.proto --grpc-gateway_out=logtostderr=true:pkg/sdk/alpha
WARNING: Package "github.com/golang/protobuf/protoc-gen-go/generator" is deprecated.
        A future release of golang/protobuf will delete this package,
        which has long been excluded from the compatibility promise.

+ protoc -I /go/src/agones.dev/agones/proto/googleapis -I /go/src/agones.dev/agones/proto/sdk/beta beta.proto --grpc-gateway_out=logtostderr=true:pkg/sdk/beta
beta.proto:20:1: warning: Import google/api/annotations.proto is unused.
WARNING: Package "github.com/golang/protobuf/protoc-gen-go/generator" is deprecated.
        A future release of golang/protobuf will delete this package,
        which has long been excluded from the compatibility promise.

This PR has many changes for migration to google.golang.org/protobuf and migration to grpc-gateway v2.

Added

1. Add proto files for proto/grpc-gateway
2. Add auto-generated files for grpc(allocation_grpc.pb.go, alpha_grpc.pb.go, beta_grpc.pb.go, sdk_grpc.pb.go, protoc-gen-openapiv2/options/annotations.pb.h, protoc-gen-openapiv2/options/openapiv2.pb.h, protoc-gen-openapiv2/annotations.pb.cc, protoc-gen-openapiv2/openapiv2.pb.cc)
3. Add grpc.gateway.protoc_gen_openapiv2.options.openapiv2_swagger to proto files in order to define swagger.json definitions
4. Add tags to each swagger files. This is tagged by protoc-gen-openapiv2
5. Add protoc-gen-openapiv2/options file's installation to sdks/cpp/CMakeLists.txt
6. Add cmake installation for protoc-gen-openapiv2 files to sdks/cpp/sources.cmake
7. Add build path for proto/grpc-gateway to sdks/rust/build.rs
8. Add description about make gen-allocation-grpc
9. Add vendor files

Changed

1. Migrade from github.com/golang/protobuf to google.golang.org/protobuf(v1.28)
2. Migrade from github.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gateway v1 to github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-grpc-gateway(v2.12.0)
3. protoc command options (e.g. --go_out=plugins=grpc is not supported. use --go-grpc_opt)
4. Use github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-openapiv2 instead of protoc-gen-swagger
5. Use google.golang.org/grpc/cmd/protoc-gen-go-grpc v1.2
6. Upgrade google.golang.org/grpc to v1.15.1
7. Upgrade grpc_release_tag to v1.15.1 for sdk-base-image
8. Upgrade grpc_release_tag to v1.15.1 for sdks/cpp/cmake/prerequisites.cmake
9. Refactor gen.sh which has difference between each language
10. Upgrade grpc-tools to 1.11.3 for nodejs sdk
11. Upgrade @grpc/grpc-js to 1.7.3 and google-protobuf to 3.21.2 for nodejs sdk
12. Change install way for protobuf from make to cmake
13. Upgrade Google.Protobuf to 3.21.9, Grpc Grpc.Core to 2.46.5, Grpc.Tools to 2.50.0 in csharp sdk
14. Update proto files under sdks/rust/proto which is copied for rust sdk
15. Change Inline streamingDefinitions in sdks/swagger/sdk.swagger.json, which is changed by protoc_gen_openapiv2(x-stream-definitions is not already supported)
16. Change test/sdk/restapi/http-api-test.go.nolint code because return code was changed
17. Change test/sdk/websocket-watch/ws-watch-test.go code because the test was not passed with the old code
18. Change tools.go dependencies
19. Change auto-generated files(alpha.pb.go, alpha.pb.gw.go, beta.pb.go, beta.pb.gw.go, sdk.pb.go, sdk.pb.gw.go, sdk.grpc.pb.h, sdk.pb.h, google/api/annotations.pb.h, google/api/http.pb.h, sdk.grpc.pb.cc, sdk.pb.cc, google/annotations.pb.cc, google/http.pb.cc, Alpha.cs, AlphaGrpc.cs, Sdk.cs, SdkGrpc.cs, alpha_grpc_pb.js, alpha_pb.js, sdk_grpc_pb.js, sdk_pb.js)
20. Change vendor files

Deleted

1. Remove github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger
2. x-stream-definitions for swagger.json is already not support. So, delete model_xstreamdefinitionssdkgameserver.go.nolint file
3. Delete vendor files

Workaround

1. Add jq logic to sdks/swagger/sdk.swagger.json because protoc-gen-openapiv2 doesn't work well in Stream and doesn't generate nessesary definitions. It seems that the bug occurs if both stream definition and --openapiv2_opt=disable_default_errors=true exist.
2. Add jq logic to build/build-allocation-images/go/gen.sh because protoc-gen-openapiv2 doesn't work well in description(even if grpc.gateway.protoc_gen_openapiv2.options.openapiv2_field is added)
3. Remove protoc-gen-openapiv2 definitions from C# sdk image because C# package doesn't support grpc-gateway

Which issue(s) this PR fixes:

Closes #2462

Special notes for your reviewer:

After #2757, I don't add the swagger client go codes to this PR for testing backward compatibility.
I'll update swagger client go codes after this PR is merged.

This is large PR. I am very careful for this change, but if you have any suggestions, please let me know.
Thank you.

[agones-bot]

Build Failed 😱

Build Id: e9c4b831-7b07-465a-b0c7-82dc4d737da8

• Cloud Build view
• Cloud Build log download

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

[agones-bot]

Build Failed 😱

Build Id: 6eae69bd-7614-4e28-bb24-71015dede517

• Cloud Build view
• Cloud Build log download

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

[agones-bot]

Build Failed 😱

Build Id: 6ba80696-4dcf-4258-843a-ebb165662e78

• Cloud Build view
• Cloud Build log download

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

[agones-bot]

Build Failed 😱

Build Id: 9cf3c289-3da5-4ca7-8c27-b41426d570cc

• Cloud Build view
• Cloud Build log download

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

[agones-bot]

Build Failed 😱

Build Id: 06dac023-42a8-4d8d-b581-628d7e68a467

• Cloud Build view
• Cloud Build log download

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

[agones-bot]

Build Failed 😱

Build Id: d07c8b4b-a4e7-456e-85cb-e8c2ce5d5870

• Cloud Build view
• Cloud Build log download

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

[agones-bot]

Build Failed 😱

Build Id: 309ff66f-d18f-4fb9-90f8-1fe31d5098b5

• Cloud Build view
• Cloud Build log download

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

[agones-bot]

Build Succeeded 👏

Build Id: 5b57e3df-70a9-4f2d-ad3c-4597eaf92b56

The following development artifacts have been built, and will exist for the next 30 days:

• image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.28.0-b0ee34e-amd64
• image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.28.0-b0ee34e-linux-amd64
• image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.28.0-b0ee34e-amd64
• image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.28.0-b0ee34e-amd64
• Linux C++ SDK (build): agonessdk-1.28.0-b0ee34e-amd64-linux-arch_64.tar.gz
• SDK Server: agonessdk-server-1.28.0-b0ee34e-amd64.zip

A preview of the website (the last 30 builds are retained):

• https://b0ee34e-dot-preview-dot-agones-images.appspot.com/

To install this version:

• git fetch https://github.com/googleforgames/agones.git pull/2786/head:pr_2786 && git checkout pr_2786
• helm install agones ./install/helm/agones --namespace agones-system --set agones.image.tag=1.28.0-b0ee34e-amd64

[markmandel]

Woah. That's a tonne of work. Well done!

I threw some questions in there, but it's all minor stuff.

I'd like @roberthbailey to also take a look, since it's a big change, make sure I didn't miss anything important.

[markmandel]

Out of curiosity, what does this do?

[govargo]

This change triggered the agones-sdk-base image build with latest grpc-version.
Without this, the CI used cache of old image and failed to compile grpc codes.

https://console.cloud.google.com/cloud-build/builds/9cf3c289-3da5-4ca7-8c27-b41426d570cc;step=16?project=agones-images

In file included from /go/src/agones.dev/agones/sdks/cpp/include/agones/sdk.h:19,
                 from /go/src/agones.dev/agones/sdks/cpp/src/agones/sdk.cc:15:
/go/src/agones.dev/agones/sdks/cpp/include/agones/sdk.grpc.pb.h:548:17: error: 'grpc::MessageAllocator' has not been declared
  548 |         ::grpc::MessageAllocator< ::agones::dev::sdk::Empty, ::agones::dev::sdk::Empty>* allocator) {
      |                 ^~~~~~~~~~~~~~~~
/go/src/agones.dev/agones/sdks/cpp/include/agones/sdk.grpc.pb.h:548:33: error: expected ',' or '...' before '<' token
  548 |         ::grpc::MessageAllocator< ::agones::dev::sdk::Empty, ::agones::dev::sdk::Empty>* allocator) {
      |                                 ^
/go/src/agones.dev/agones/sdks/cpp/include/agones/sdk.grpc.pb.h:575:17: error: 'grpc::MessageAllocator' has not been declared
  575 |         ::grpc::MessageAllocator< ::agones::dev::sdk::Empty, ::agones::dev::sdk::Empty>* allocator) {
      |                 ^~~~~~~~~~~~~~~~
/go/src/agones.dev/agones/sdks/cpp/include/agones/sdk.grpc.pb.h:575:33: error: expected ',' or '...' before '<' token
  575 |         ::grpc::MessageAllocator< ::agones::dev::sdk::Empty, ::agones::dev::sdk::Empty>* allocator) {
      |                                 ^
...

-DCMAKE_BUILD_TYPE=Release is needed for install grpc and its dependency, I think.
I referred to the official sample.

https://console.cloud.google.com/cloud-build/builds/d07c8b4b-a4e7-456e-85cb-e8c2ce5d5870;step=16?project=agones-images

-- Could NOT find absl (missing: absl_DIR)
-- Configuring done
CMake Error at CMakeLists.txt:130 (add_library):
  Target "agones" links to target "absl::flat_hash_map" but the target was
  not found.  Perhaps a find_package() call is missing for an IMPORTED
  target, or an ALIAS target is missing?
...

[markmandel]

Oh got it!

You could have also updated

Base images for SDKs Bump: 2

To

Base images for SDKs Bump: 3

Just to update the file, which resets the build, but no reason to fix I don't think.

[govargo]

Oh I didn't know # Base images for SDKs Bump: XX!!!
After another review, if I should revert -DCMAKE_BUILD_TYPE=Release, please let me know.

[govargo]

My apologies for my late reply, I was on a trip last week and had work for that period.
I'll update this PR.
It may take some time to confirm the local test, but I'll update withn this week.

[markmandel]

My apologies for my late reply, I was on a trip last week and had work for that period. I'll update this PR. It may take some time to confirm the local test, but I'll update withn this week.

All good! Thanks for doing all this work!

[agones-bot]

Build Succeeded 👏

Build Id: 15337cbd-7e79-4a29-abab-7eadec7a404d

The following development artifacts have been built, and will exist for the next 30 days:

• image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.28.0-fe328fd-amd64
• image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.28.0-fe328fd-linux-amd64
• image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.28.0-fe328fd-amd64
• image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.28.0-fe328fd-amd64
• Linux C++ SDK (build): agonessdk-1.28.0-fe328fd-amd64-linux-arch_64.tar.gz
• SDK Server: agonessdk-server-1.28.0-fe328fd-amd64.zip

A preview of the website (the last 30 builds are retained):

• https://fe328fd-dot-preview-dot-agones-images.appspot.com/

To install this version:

• git fetch https://github.com/googleforgames/agones.git pull/2786/head:pr_2786 && git checkout pr_2786
• helm install agones ./install/helm/agones --namespace agones-system --set agones.image.tag=1.28.0-fe328fd-amd64

[agones-bot]

Build Succeeded 👏

Build Id: 0ad046f8-0024-4ddd-84cd-8bab2efb30dd

The following development artifacts have been built, and will exist for the next 30 days:

• image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.28.0-d7a33eb-amd64
• image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.28.0-d7a33eb-linux-amd64
• image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.28.0-d7a33eb-amd64
• image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.28.0-d7a33eb-amd64
• Linux C++ SDK (build): agonessdk-1.28.0-d7a33eb-amd64-linux-arch_64.tar.gz
• SDK Server: agonessdk-server-1.28.0-d7a33eb-amd64.zip

A preview of the website (the last 30 builds are retained):

• https://d7a33eb-dot-preview-dot-agones-images.appspot.com/

To install this version:

• git fetch https://github.com/googleforgames/agones.git pull/2786/head:pr_2786 && git checkout pr_2786
• helm install agones ./install/helm/agones --namespace agones-system --set agones.image.tag=1.28.0-d7a33eb-amd64

[markmandel]

This looks good to me! No issues on my end.

@roberthbailey did you have a chance to look over this, since it's so big?

@zmerlynn would also love your eyes as well. I figure more reviews the better.

[markmandel]

Noice.

[markmandel]

Oh got it!

You could have also updated

Base images for SDKs Bump: 2

To

Base images for SDKs Bump: 3

Just to update the file, which resets the build, but no reason to fix I don't think.

[zmerlynn]

This LGTM, but maybe we wait until RC is cut and then merge? That'll give it the whole next release to shake out.

[markmandel]

This LGTM, but maybe we wait until RC is cut and then merge? That'll give it the whole next release to shake out.

+1. I like this idea, but merge after 1.28 release is stable and out. So it has all of the 1.29 dev time to bake.

[agones-bot]

Build Succeeded 👏

Build Id: e708a7e3-5098-478e-b61a-63a2919c1be9

The following development artifacts have been built, and will exist for the next 30 days:

• image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.28.0-e9b60ea-amd64
• image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.28.0-e9b60ea-linux-amd64
• image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.28.0-e9b60ea-amd64
• image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.28.0-e9b60ea-amd64
• Linux C++ SDK (build): agonessdk-1.28.0-e9b60ea-amd64-linux-arch_64.tar.gz
• SDK Server: agonessdk-server-1.28.0-e9b60ea-amd64.zip

A preview of the website (the last 30 builds are retained):

• https://e9b60ea-dot-preview-dot-agones-images.appspot.com/

To install this version:

• git fetch https://github.com/googleforgames/agones.git pull/2786/head:pr_2786 && git checkout pr_2786
• helm install agones ./install/helm/agones --namespace agones-system --set agones.image.tag=1.28.0-e9b60ea-amd64

[markmandel]

[markmandel]

Any objections to labelling this breaking just so we remember to note in our release notes as "hey, handle this upgrade with care" ?

[agones-bot]

Build Succeeded 👏

Build Id: d460a383-bfa2-47d7-bfbc-1bd80ed2be10

The following development artifacts have been built, and will exist for the next 30 days:

• image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.29.0-235c283-amd64
• image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.29.0-235c283-linux-amd64
• image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.29.0-235c283-amd64
• image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.29.0-235c283-amd64
• Linux C++ SDK (build): agonessdk-1.29.0-235c283-amd64-linux-arch_64.tar.gz
• SDK Server: agonessdk-server-1.29.0-235c283-amd64.zip

A preview of the website (the last 30 builds are retained):

• https://235c283-dot-preview-dot-agones-images.appspot.com/

To install this version:

• git fetch https://github.com/googleforgames/agones.git pull/2786/head:pr_2786 && git checkout pr_2786
• helm install agones ./install/helm/agones --namespace agones-system --set agones.image.tag=1.29.0-235c283-amd64

[zmerlynn]

Any objections to labelling this breaking just so we remember to note in our release notes as "hey, handle this upgrade with care" ?

Hmm, if the protobufs are not wire compatible I would be worried. Do we have skew testing of any sort?

Otherwise this feels wholly internal and I'm not sure we need to be super careful?

[markmandel]

I mean it should be fine, but like when uupgrading gRPC versions, I'm always like "definitely make sure to test release in dev, not in prod" 😄

[markmandel]

This also includes a major version update of grpc-gateway. We've backward comparability tested as much as possible, but I figure good to give people the heads up.

[zmerlynn]

This also includes a major version update of grpc-gateway. We've backward comparability tested as much as possible, but I figure good to give people the heads up.

SGTM.

[markmandel]

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: govargo, markmandel

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [markmandel]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[govargo]

It looks like this was merged to my repository(govargo/agones)'s main branch. https://github.com/govargo/agones/commits/migrate-protobuf

Could you check this again?

Thanks in advance.

[zmerlynn]

@govargo I believe that was @roberthbailey catching the branch up to main. This PR still needs to be merged.

[govargo]

Thank you all!