grpc: switch servers and retry on error

[boxofrad]

Description

This is the OSS portion of enterprise PR 3822, it has been reviewed thoroughly there.

It adds a custom gRPC balancer that replicates the router's server cycling behavior. It also enables automatic retries for RESOURCE_EXHAUSTED errors, which we now get for free.

Testing & Reproduction steps

The balancer package has unit tests that spin up real gRPC servers and clients.

Also manually tested in enterprise by:

• Hacking the partition read endpoint to return RESOURCE_EXHAUSTED if req.Name != NodeName
• Running two servers and a client agent
• Running consul partition read server1 and consul partition read server2
• Watching the agent logs and observing:

2022-12-09T12:15:45.850Z [TRACE] agent.grpc.balancer: witnessed RPC error: target=consul://dc1.7105466e-b7d7-8240-a500-88a14461aa49/server.dc1 server=dc1-127.0.0.1:9102 error="rpc error: code = ResourceExhausted desc = you got rate limited, my dude"
2022-12-09T12:15:45.850Z [DEBUG] agent.grpc.balancer: switching server: target=consul://dc1.7105466e-b7d7-8240-a500-88a14461aa49/server.dc1 from=dc1-127.0.0.1:9102 to=dc1-127.0.0.1:9101
2022-12-09T12:15:45.850Z [TRACE] agent.grpc.balancer: sub-connection state changed: target=consul://dc1.7105466e-b7d7-8240-a500-88a14461aa49/server.dc1 server=dc1-127.0.0.1:9101 state=CONNECTING
2022-12-09T12:15:45.851Z [TRACE] agent.grpc.balancer: sub-connection state changed: target=consul://dc1.7105466e-b7d7-8240-a500-88a14461aa49/server.dc1 server=dc1-127.0.0.1:9101 state=READY
2022-12-09T12:16:12.049Z [ERROR] agent.http: Request error: method=GET url=/v1/partition/foo from=127.0.0.1:62096 error="Partition not found for \"server2\""

Note that the balancer automatically switched connections and retried against the other server 🙌🏻

[boxofrad]

Hey @jmurret 👋🏻

Not sure if you've seen one of these manual ENT→OSS PRs before? Shout if you need any direction 🙇🏻