Consider migration from gogo to vtprotobuf

[alexshtin]

Currently we use https://github.com/golang/protobuf which is stable release. There is another repo under active development https://github.com/protocolbuffers/protobuf-go which is considered to be "Next Generation Go Protocol Buffers". I found number of improvements there but don't want to use it right away because it states:

WARNING: This repository is in active development. There are no guarantees about API stability. Breaking changes will occur until a stable release is made and announced.

[alexshtin]

It is released: https://blog.golang.org/a-new-go-api-for-protocol-buffers but gRPC support is not there.

[alexshtin]

Consider switching to https://github.com/golang/protobuf/releases/tag/v1.4.0-rc.4 at least.

[alexshtin]

Discussion about future gRPC support in v2: golang/protobuf#1070.

[samarabbas]

@alexshtin is this still relevant?

[alexshtin]

It might be relevant one day. We are not using Google protobuf/gRPC plugin but using gogo. When v2 was finally released I made some tests and found out that gogo implementation still outperform it significantly. In a meantime gogo team said that adoption of v2 would require a lot of work and abandoned the project.

Recently vitess team created their own compiler which has some limitations but looks very promising. I would give it some time to mature and then investigate how hard it would be for us to migrate to it.

[alexshtin]

Other options to consider:

We have implemented fast marshaling and fast proto reflection for the golang proto v2 API. There are a few additional features in the works for our use case similar to customtype that may or may not be desirable for other protobuf users: https://github.com/cosmos/cosmos-proto

We’ve added a few more features to protopatch in the last few weeks, like support for embedded structs and optional: https://github.com/alta/protopatch

[alexshtin]

One more option: https://github.com/CrowdStrike/csproto

[alexshtin]

Vitess Protocol Buffers compiler: https://github.com/planetscale/vtprotobuf.

[GiantRobots]

Can we also look into using buf's Connect as well?

https://connect.build/docs/go/getting-started

Their connect protocol would be incredibly nice for the frontend (human readable network format so we can use built in browser tools) and it would entirely remove the need for ui-server as the clients they generate can talk directly to browsers through the connect protocol or the grpc-web protocol.

https://connect.build/docs/go/grpc-compatibility#migration

[akshayjshah]

Hey! I work at Buf, designed Connect, and used to work on one of Cadence's old transports (tchannel) too. It's a small world :)

We're very familiar with the pain of migrating off gogoprotobuf. The best alternative available today for performance-conscious Go programmers is vtprotobuf, which generates some extra code that works with the google.golang.org/protobuf standard code gen. It's pretty easy to integrate with grpc-go. The end result still isn't as fast as gogoprotobuf, but it's the best you can get with packages that are still maintained.

csprotobuf helped CrowdStrike migrate off gogoprotobuf - it's basically a shim layer that lets your system continue to function mid-migration. The author, Dylan Bourque, is often in the protobuf and grpc channels in the Gophers Slack. It may help when/if you decide to take on this work.

@GiantRobots, Connect is largely orthogonal to this discussion - it'll work with any of the protobuf runtimes above. There are a few ways you can migrate your frontend to use connect-es and the Connect RPC protocol without migrating your backend off grpc-go + gogo. One relatively easy option is to replace temporal-ui with Envoy and enable Connect-to-gRPC protocol translation (or enable that filter in an existing Envoy), and register a gRPC codec that uses gogo's jsonpb package in your backend. The change to your backend will be just a few lines of code and won't affect any business logic, and Envoy will work without any code generation or schema access.

[alexshtin]

After many discussions and considering different approaches we decided to migrate to the standard Google library. This migration is already completed, mostly in #5032 and few followed PRs.