embed: add `GRPCAdditionalServerOptions` config #14066

rleungx · 2022-05-23T10:58:13Z

This PR introduces GRPCAdditionalServerOptions which allow changing the internal gRPC settings. Sometimes, we may register our own gRPC service into etcd and change the max-request-bytes might affect the internal etcd logic.

codecov-commenter · 2022-05-24T04:05:12Z

Codecov Report

Merging #14066 (5d6a29d) into main (a1405e9) will decrease coverage by 0.33%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main   #14066      +/-   ##
==========================================
- Coverage   75.60%   75.27%   -0.34%     
==========================================
  Files         457      457              
  Lines       37084    37117      +33     
==========================================
- Hits        28038    27939      -99     
- Misses       7312     7414     +102     
- Partials     1734     1764      +30

Flag	Coverage Δ
all	`75.27% <100.00%> (-0.34%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
server/embed/config.go	`73.40% <ø> (ø)`
server/embed/etcd.go	`75.38% <100.00%> (+0.04%)`	⬆️
server/etcdserver/api/membership/store.go	`50.00% <0.00%> (-10.00%)`	⬇️
server/storage/mvcc/watchable_store.go	`84.42% <0.00%> (-8.34%)`	⬇️
client/pkg/v3/fileutil/purge.go	`66.03% <0.00%> (-7.55%)`	⬇️
api/v3rpc/rpctypes/error.go	`84.61% <0.00%> (-5.87%)`	⬇️
client/v3/experimental/recipes/key.go	`75.34% <0.00%> (-5.48%)`	⬇️
server/lease/lease.go	`94.87% <0.00%> (-5.13%)`	⬇️
raft/rafttest/node.go	`95.00% <0.00%> (-5.00%)`	⬇️
client/v3/leasing/cache.go	`87.77% <0.00%> (-3.89%)`	⬇️
... and 19 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

ahrtr · 2022-05-25T06:36:25Z

Comments:

What do you mean by Sometimes, we may register our own gRPC service into etcd? Could you explain this?
Why you do not expose the two items? i.e. config.go#L132

rleungx · 2022-05-25T07:03:46Z

Comments:

What do you mean by Sometimes, we may register our own gRPC service into etcd? Could you explain this?

Why you do not expose the two items? i.e. config.go#L132

See

etcd/server/embed/config.go

Lines 302 to 309 in bfb9aa4

    
           // ServiceRegister is for registering users' gRPC services. A simple usage example: 
        
           //	cfg := embed.NewConfig() 
        
           //	cfg.ServerRegister = func(s *grpc.Server) { 
        
           //		pb.RegisterFooServer(s, &fooServer{}) 
        
           //		pb.RegisterBarServer(s, &barServer{}) 
        
           //	} 
        
           //	embed.StartEtcd(cfg) 
        
           ServiceRegister func(*grpc.Server) `json:"-"`

I can also expose them if you want.

ahrtr · 2022-05-27T02:28:03Z

server/etcdserver/api/v3rpc/grpc.go

@@ -66,8 +62,8 @@ func Server(s *etcdserver.EtcdServer, tls *tls.Config, interceptor grpc.UnarySer
 	opts = append(opts, grpc.UnaryInterceptor(grpc_middleware.ChainUnaryServer(chainUnaryInterceptors...)))
 	opts = append(opts, grpc.StreamInterceptor(grpc_middleware.ChainStreamServer(chainStreamInterceptors...)))

-	opts = append(opts, grpc.MaxRecvMsgSize(int(s.Cfg.MaxRequestBytes+grpcOverheadBytes)))
-	opts = append(opts, grpc.MaxSendMsgSize(maxSendBytes))
+	opts = append(opts, grpc.MaxRecvMsgSize(s.Cfg.MaxRecvMsgSize))


What if MaxRequestBytes is different from s.Cfg.MaxRecvMsgSize? If you configure a big value for s.Cfg.MaxRecvMsgSize, i.e. 10MB, while the MaxRequestBytes is 1 MB, then etcdserver may still reject the request.

Do you mean MaxRecvMsgSize should be larger or equal to MaxRequestBytes?

I don't see any reason to add the two parameters.

Did you see any issue without adding MaxRecvMsgSize/MaxSendMsgSize?

Actually, we have a feature that needs to change MaxRecvMsgSize to 150M. I was wondering if etcd can offer a separate config to customize it without changing MaxRequestBytes. For now, when we start a cluster, it will give a warning about the MaxRequestBytes exceeds the recommended size.

A couple of comments:

etcd is designed to serve small metadata. Why do you need to change MaxRecvMsgSize to 150M? Can you elaborate your use case?

Have you verified your case based on this PR with MaxRecvMsgSize being set to 150M, while keep --max-request-bytes unchanged?

Any update on this?

Sorry for the late reply.

etcd is designed to serve small metadata. Why do you need to change MaxRecvMsgSize to 150M? Can you elaborate your use case?

We will register our service into the embed etcd. See https://github.com/tikv/pd/blob/0d05fba64372408c5af9a78cd9b47b58c4925ca9/server/server.go#L277-L280. For example, the store heartbeat https://github.com/tikv/pd/blob/0d05fba64372408c5af9a78cd9b47b58c4925ca9/server/grpc_service.go#L570.When we need to recover our cluster, the heartbeat request will bring the meta of data, which could be large. We utilize the meta to decide which part of the data should be recovered.

Have you verified your case based on this PR with MaxRecvMsgSize being set to 150M, while keep --max-request-bytes unchanged?

not yet

I would consider registering a hook to alter the grpc options as part of the config:

func additionalGrpcServerOptions() grpc.ServerOption[]

Actually:

grpcAdditionalServerOptions grpc.ServerOption[]

in the config would do the trick.

ptabor

grpcAdditionalServerOptions grpc.ServerOption[]

ptabor · 2022-06-13T08:47:44Z

server/config/config.go

@@ -129,6 +129,11 @@ type ServerConfig struct {
 	// MaxRequestBytes is the maximum request size to send over raft.
 	MaxRequestBytes uint

+	// MaxRecvMsgSize is the max gRPC message size in bytes the server can receive.


How about adding
grpcAdditionalServerOptions grpc.ServerOption[] as a generic options instead ?

ahrtr · 2022-06-13T11:00:02Z

@ptabor One item in my to do list is to sort out all the thresholds in etcd, including this one. Will discuss this in a separate session once my current task of refactoring lease is done.

ptabor · 2022-06-14T20:27:30Z

@ptabor One item in my to do list is to sort out all the thresholds in etcd, including this one. Will discuss this in a separate session once my current task of refactoring lease is done.

@ahrtr As this sound as a snowflake use-case (grpc shared between etcd and other service), I think we should not be in the business of looking at the limits... and allow programmers (only in embed server) to pass additional grpc settings, that etcd passes-through.

ahrtr · 2022-06-15T01:42:01Z

@ahrtr As this sound as a snowflake use-case (grpc shared between etcd and other service), I think we should not be in the business of looking at the limits... and allow programmers (only in embed server) to pass additional grpc settings, that etcd passes-through.

Overall adding a generic grpcAdditionalServerOptions grpc.ServerOption[] looks good to me. But it doesn't resolve the concern that it may have impact on the build-in services provided by etcd itself.

The good side is flexibility, because users can set any additional options they want. The down side is it may have impact on the gRPC service provided by etcd itself.

Another perspective to think about this is that we should take all services, including etcd build-in services and users registered services, under control. Once users see any issue for the embedded etcd, they will also raise issue in etcd community. Based on the principle of equal responsibility and obligation, If we need to support such case, then we should can add more restriction on the services registered by users, instead of just providing flexibility without any restriction. So users shouldn't set any additional options, instead they should follow all the existing options from this perspective.

ahrtr · 2022-06-15T02:06:16Z

The new feature of ServiceRegister was introduced in pull/7215.

Again, the good side of the PR is users can reuse the same port as etcd. But the bad side is it introduces additional unnecessary complexity to etcd. Personally I don't like the PR ( I mean 7215), and the better choice should be resolving it out of the box (such as providing a proxy/gateway in front of embedded etcd and users' gRPC service) instead of hacking etcd itself.

rleungx · 2022-06-15T04:03:27Z

@ahrtr As this sound as a snowflake use-case (grpc shared between etcd and other service), I think we should not be in the business of looking at the limits... and allow programmers (only in embed server) to pass additional grpc settings, that etcd passes-through.

Overall adding a generic grpcAdditionalServerOptions grpc.ServerOption[] looks good to me. But it doesn't resolve the concern that it may have impact on the build-in services provided by etcd itself.

The good side is flexibility, because users can set any additional options they want. The down side is it may have impact on the gRPC service provided by etcd itself.

Another perspective to think about this is that we should take all services, including etcd build-in services and users registered services, under control. Once users see any issue for the embedded etcd, they will also raise issue in etcd community. Based on the principle of equal responsibility and obligation, If we need to support such case, then we should can add more restriction on the services registered by users, instead of just providing flexibility without any restriction. So users shouldn't set any additional options, instead they should follow all the existing options from this perspective.

IMO, no matter if it's a requirement of user-defined services, we still need to provide the ability to change the default gRPC settings.

rleungx · 2022-06-15T10:04:02Z

Besides, supporting the custom interceptor for embed etcd is another need. See #13468

ahrtr · 2022-06-17T22:06:03Z

Note that the users registered service and the etcd build-in gRPC services share the same gRPC channel. Any options will have impact on both of them.

Please just use the existing flag --max-request-bytes to change the MaxRecvSize. Do you need to change the MaxSendSize in your case?

nolouch · 2022-06-20T13:03:06Z

Hi @ahrtr
#7215 not allow users to define gRPC's setting, I think this PR is to improve it.

We have some requirements with the embed etcd use-case, not only max-request-bytes :

register own service in etcd, and we want to protect with those services, but etcd's service may influence it (some client directly use etcd service), so we need QoS features. But same as the issue comment says, it suggests this can be simply implemented as a gRPC server middleware, and keep etcd as minimal as possible, but we cannot use custom gRPC server middleware
Fixed parameters bring some inconvenience because it is not extensible, we need to add supported parameters to etcd first, and then wait for it release to the stable version, and then update the dependencies. This time period is too long.

IMO, embed etcd users need to be responsible for changing etcd parameters, this does not affect normal etcd users(standalone process). So hopefully this PR will be approved, it is more convenient for embed etcd users. ptal @ahrtr @ptabor, Thanks.

nolouch · 2022-06-20T13:04:41Z

server/embed/etcd.go

@@ -719,6 +719,7 @@ func (e *Etcd) serveClients() (err error) {
 			Time:    e.cfg.GRPCKeepAliveInterval,
 			Timeout: e.cfg.GRPCKeepAliveTimeout,
 		}))
+		gopts = append(gopts, e.cfg.GRPCAdditionalServerOptions...)


Does it need to move out this if statement?

ptabor · 2022-06-22T10:00:19Z

server/embed/etcd.go

@@ -719,6 +719,7 @@ func (e *Etcd) serveClients() (err error) {
 			Time:    e.cfg.GRPCKeepAliveInterval,
 			Timeout: e.cfg.GRPCKeepAliveTimeout,
 		}))
+		gopts = append(gopts, e.cfg.GRPCAdditionalServerOptions...)


ptabor · 2022-06-22T10:06:36Z

tests/framework/integration/cluster.go

-	GrpcURL        string
-	GrpcBridge     *bridge
+	GrpcServerOpts           []grpc.ServerOption
+	GrpcAdditionalServerOpts []grpc.ServerOption


Do we really need it here ?

I think that MustNewMember could just propagate MemberConfig.GrpcAdditionalServerOpts into GrpcServerOpts and not store it here explicitly.

ahrtr · 2022-06-22T11:37:31Z

Hi @ahrtr #7215 not allow users to define gRPC's setting, I think this PR is to improve it.

We have some requirements with the embed etcd use-case, not only max-request-bytes :
* register own service in etcd, and we want to protect with those services, but etcd's service may influence it (some client directly use etcd service), so we need `QoS features`. But same as the [issue comment](https://github.com/etcd-io/etcd/pull/12290#issuecomment-697060195) says, it suggests this can be simply implemented as a gRPC server middleware, and keep etcd as minimal as possible, but we cannot use custom gRPC server middleware

* Fixed parameters bring some inconvenience because it is not extensible, we need to add supported parameters to etcd first, and then wait for it release to the stable version, and then update the dependencies. This time period is too long.
IMO, embed etcd users need to be responsible for changing etcd parameters, this does not affect normal etcd users(standalone process). So hopefully this PR will be approved, it is more convenient for embed etcd users. ptal @ahrtr @ptabor, Thanks.

I am still not convinced that we really need to add grpcAdditionalServerOptions grpc.ServerOption[]. I am not saying we shouldn't add it, instead I am just trying to say that we shouldn't add any new feature without a deep understanding the use cases beforehand. So please elaborate your use cases firstly. Why do you need it, and how will use it after adding it? Specifically, what gRPC options are you going to add via the grpcAdditionalServerOptions?

Note that there are several QoS related PRs, but none of them is merged so far. If your discussion is based on the QoS feature, then it isn't valid at the moment.

More flexibility also means more chance to make mistake. If there is no any strong incentive or reasonable benefit, then we shouldn't add it.

Please also make sure you well understood my previous two comments, issuecomment-1155881593 and issuecomment-1155894742

ptabor · 2022-06-22T21:51:57Z

@ahrtr:

I'm not insisting on merging, this... just steering towards such variant rather than explosion of custom options.
A use-case is linked from: embed: add GRPCAdditionalServerOptions config #14066 (comment), and seems direct consequence of allowing embed server to have custom customer's services being used. I think usage of embedded service put's a high bar on customer to have proper testing for their integrated solution. We should document this requirement more explicitly.
I was hoping (but seems not possible currently) that we could perform validation of different constants/limit post-grpc server start, It's run it and obtain the effective config. Unfortunately grpc does not exposes the resolved configuration of running server publicly:

https://github.com/grpc/grpc-go/blob/28de4866ce7440b675662abbdd5c43b476bd4dae/server.go#L124

ahrtr · 2022-06-23T01:50:59Z

Thanks @ptabor for the feedback.

I read the 14066#discussion_r887444999 before. etcd has already supported users registering custom service. If users need to configure the max request size, we already have MaxRequestBytes (MaxRecvMsgSize). If users need MaxSendMsgSize as well, we can add it (the default value is math.MaxInt32, do we really need to expose it?). Apart from that, are there any other options users might need?

FYI. there is a on-going PR 14081 for the MaxConcurrentStreams.

There are 24 options in total which are supported by gRPC 1.47 for now, see server.go#L143-L168 . Currently etcd exposes about half of them. Some options/API (such as InTapHandle, ConnectionTimeout, HeaderTableSize and NumStreamWorkers) are still experimental, which probably I don't think are proper to expose to etcd users. For other options (such as RPCCompressor, RPCDecompressor, InitialWindowSize, InitialConnWindowSize, WriteBufferSize and ReadBufferSize) which etcd hasn't exposed yet, I am not sure whether we should expose them. Personally I don't think it's good to just expose them all without deep understanding and full testing ourselves (the community) beforehand, and we do not receive real request on these so far.

I'm not insisting on merging, this... just steering towards such variant rather than explosion of custom options.

The number of options will not explode. Please see my comment above.

I think usage of embedded service put's a high bar on customer to have proper testing for their integrated solution. We should document this requirement more explicitly.

This might be a valid point.

rleungx · 2022-06-23T04:05:11Z

@ahrtr @ptabor Thanks for your reply. #14066 (comment) make sense to me. But for now, we indeed have two requirements:

support changing the MaxRecvMsgSize which can be replaced by changing max-request-bytes. It doesn't well conform to semantics but it works. I'm ok with the current usage.
support adding the custom gRPC interceptor for user-defined gRPC services so we can add our logic to intercept the request and do something, e.g. the QoS feature or forwarding requests. But etcd doesn't support it at the moment.

nolouch · 2022-06-23T06:46:19Z

@ahrtr

Once users see any issue for the embedded etcd, they will also raise issue in etcd community. Based on the principle of equal responsibility and obligation, If we need to support such case, then we should can add more restrictions on the services registered by users, instead of just providing flexibility without any restriction.

I don't think it's good to just expose them all without deep understanding and full testing ourselves (the community) beforehand

I have some questions:

Community can issue that and add the option if approve, as I said before, users need to wait a long time. on the other hand, how do restriction users not use the inappropriate option value that may affect etcd?
As you said, currently etcd exposes about half of 24 options, so about 12 options. Do all users really deep understand those 12 options and full testing by yourselves with all use cases?

I think there still have the same problems. so if the embed user wants to change the default parameters, they should understand it, and be responsible for it. that means users should test it and make sure it is suitable for themselves scenarios. As @ptabor said, we can document it.
And I don't think it introduces unnecessary complexity to etcd, etcd still keep minimal, and it's more flexibility, let user can extend etcd (such as grpc services and middlewares), and also it only affects embed users.

nolouch · 2022-06-23T07:08:33Z

BTW, If you really concert about it, I think etcd embed model should be more like a library, it can be as a service be registered to a custom grpc server rather than let itself as a server.

ahrtr · 2022-08-08T21:41:15Z

I am OK to revisit this PR, but it needs at least three maintainers' approval.

ahrtr · 2024-06-06T08:34:57Z

Friendly ping @ahrtr

Sorry for the late response. Please rebase this PR.

rleungx · 2024-06-07T08:37:07Z

Friendly ping @ahrtr

Sorry for the late response. Please rebase this PR.

done

Signed-off-by: Ryan Leung <rleungx@gmail.com>

ahrtr · 2024-06-08T12:36:00Z

server/embed/config.go

+	// GRPCAdditionalServerOptions is the additional server option hook
+	// for changing the default internal gRPC configuration.


Suggested change

// GRPCAdditionalServerOptions is the additional server option hook

// for changing the default internal gRPC configuration.

// GRPCAdditionalServerOptions is the additional server option hook

// for changing the default internal gRPC configuration. Note these

// additional configurations take precedence over the existing individual

// configurations if present. Please refer to

// https://github.com/etcd-io/etcd/pull/14066#issuecomment-1248682996

ahrtr · 2024-06-08T12:39:49Z

Overall looks good to me. cc @fuweid @jmhbnz @serathius @tjungblu

tjungblu · 2024-06-10T07:18:15Z

same. /lgtm

Signed-off-by: Ryan Leung <rleungx@gmail.com>

ahrtr · 2024-06-12T05:03:37Z

@rleungx FYI. #18015 (comment)

rleungx · 2024-06-12T07:11:28Z

@rleungx FYI. #18015 (comment)

Got it.

rleungx · 2024-06-14T03:18:20Z

@ahrtr Is there anything I need to do to get this PR merged?

ahrtr · 2024-06-14T05:28:20Z

server/embed/config.go

+	// GRPCAdditionalServerOptions is the additional server option hook
+	// for changing the default internal gRPC configuration. Note these
+	// additional configurations take precedence over the existing individual
+	// configurations if present. Please refer to
+	// https://github.com/etcd-io/etcd/pull/14066#issuecomment-1248682996
+	GRPCAdditionalServerOptions []grpc.ServerOption `json:"grpc-additional-server-options"`


It would be better if we also add a page under the https://etcd.io/docs/v3.5/dev-guide/ to document this. Can you raise a ticket in https://github.com/etcd-io/website/issues?

etcd-io/website#862

ahrtr · 2024-06-14T05:28:52Z

cc @fuweid @ivanvc @jmhbnz @serathius

ivanvc

I've been reviewing this pull request back and forth. The implementation looks good. But I see some previous discussion about whether this is a feature we want to bring or not. It seems like the latest is that there is a consensus. Maybe it would be good to hear from other team members, too.

jmhbnz

Thanks for the work on this @rleungx.

Given this is a more complex configuration option I suggest adding a commented out example in https://github.com/etcd-io/etcd/blob/main/etcd.conf.yml.sample showing how this option could be configured via configuration file if desired.

ahrtr · 2024-06-19T04:39:41Z

Thanks for the work on this @rleungx.

Given this is a more complex configuration option I suggest adding a commented out example in https://github.com/etcd-io/etcd/blob/main/etcd.conf.yml.sample showing how this option could be configured via configuration file if desired.

The new option is only for the embedded use case. Users won't be able to configure it through the command line or config file.

jmhbnz

Thanks for clarification @ahrtr.

LGTM

ahrtr · 2024-06-19T05:32:51Z

Please also update changelog-3.6, which can be done in a separate PR. @rleungx

rleungx · 2024-06-19T08:41:44Z

Please also update changelog-3.6, which can be done in a separate PR. @rleungx

Sure. Once it is merged, I will raise another PR to update it.

okJiang · 2024-08-07T10:07:23Z

@ahrtr Sorry to bother you, can this PR accept a pick back to 3.5? I can submit a PR for review. Any suggestions are valuable to me.

ahrtr · 2024-08-07T11:00:56Z

@ahrtr Sorry to bother you, can this PR accept a pick back to 3.5? I can submit a PR for review. Any suggestions are valuable to me.

We may not backport this (feature) to stable release without strong justification.

rleungx marked this pull request as ready for review May 23, 2022 10:58

rleungx force-pushed the add-config branch from b41efb4 to 460b71f Compare May 23, 2022 11:19

ahrtr reviewed May 27, 2022

View reviewed changes

ptabor suggested changes Jun 13, 2022

View reviewed changes

rleungx changed the title ~~embed: add MaxRecvMsgSize/MaxSendMsgSize config~~ embed: add GRPCAdditionalServerOptions config Jun 15, 2022

ahrtr mentioned this pull request Jun 15, 2022

Restriction on MaxRequestSize and maxWALEntrySize #14114

Closed

nolouch reviewed Jun 20, 2022

View reviewed changes

ptabor reviewed Jun 22, 2022

View reviewed changes

rleungx force-pushed the add-config branch 4 times, most recently from 9f9ce3f to 6a6b493 Compare August 9, 2022 06:14

stale bot removed the stale label Jun 6, 2024

k8s-ci-robot added the area/testing label Jun 6, 2024

rleungx force-pushed the add-config branch from 962d4ca to bd79efc Compare June 7, 2024 08:36

introduce GRPCAdditionalServerOptions

29abd62

Signed-off-by: Ryan Leung <rleungx@gmail.com>

rleungx force-pushed the add-config branch from bd79efc to 29abd62 Compare June 7, 2024 08:39

rleungx requested a review from ahrtr June 7, 2024 08:42

ahrtr reviewed Jun 8, 2024

View reviewed changes

address the comment

d9cb8b8

Signed-off-by: Ryan Leung <rleungx@gmail.com>

k8s-ci-robot added the size/M label Jun 12, 2024

ahrtr reviewed Jun 14, 2024

View reviewed changes

rleungx mentioned this pull request Jun 18, 2024

Document the embed etcd configurations and their usage etcd-io/website#862

Open

ivanvc approved these changes Jun 18, 2024

View reviewed changes

jmhbnz reviewed Jun 18, 2024

View reviewed changes

jmhbnz approved these changes Jun 19, 2024

View reviewed changes

ahrtr merged commit 1d13fc5 into etcd-io:main Jun 21, 2024
40 checks passed

rleungx mentioned this pull request Jun 21, 2024

Update release-3.6 changelog #18210

Merged

rleungx deleted the add-config branch June 21, 2024 09:09

		// GRPCAdditionalServerOptions is the additional server option hook
		// for changing the default internal gRPC configuration.

embed: add GRPCAdditionalServerOptions config #14066

embed: add GRPCAdditionalServerOptions config #14066

Conversation

rleungx commented May 23, 2022 • edited Loading

codecov-commenter commented May 24, 2022 • edited Loading

Codecov Report

ahrtr commented May 25, 2022

rleungx commented May 25, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ptabor left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ahrtr commented Jun 13, 2022

ptabor commented Jun 14, 2022

ahrtr commented Jun 15, 2022 • edited Loading

ahrtr commented Jun 15, 2022

rleungx commented Jun 15, 2022

rleungx commented Jun 15, 2022

ahrtr commented Jun 17, 2022

nolouch commented Jun 20, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ahrtr commented Jun 22, 2022 • edited Loading

ptabor commented Jun 22, 2022

ahrtr commented Jun 23, 2022 • edited Loading

rleungx commented Jun 23, 2022

nolouch commented Jun 23, 2022 • edited Loading

nolouch commented Jun 23, 2022 • edited Loading

ahrtr commented Aug 8, 2022

ahrtr commented Jun 6, 2024

rleungx commented Jun 7, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ahrtr commented Jun 8, 2024

tjungblu commented Jun 10, 2024

ahrtr commented Jun 12, 2024

rleungx commented Jun 12, 2024

rleungx commented Jun 14, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ahrtr commented Jun 14, 2024

ivanvc left a comment

Choose a reason for hiding this comment

jmhbnz left a comment

Choose a reason for hiding this comment

ahrtr commented Jun 19, 2024

jmhbnz left a comment

Choose a reason for hiding this comment

ahrtr commented Jun 19, 2024

rleungx commented Jun 19, 2024

okJiang commented Aug 7, 2024

ahrtr commented Aug 7, 2024

embed: add `GRPCAdditionalServerOptions` config #14066

embed: add `GRPCAdditionalServerOptions` config #14066

rleungx commented May 23, 2022 •

edited

Loading

codecov-commenter commented May 24, 2022 •

edited

Loading

ahrtr commented Jun 15, 2022 •

edited

Loading

nolouch commented Jun 20, 2022 •

edited

Loading

ahrtr commented Jun 22, 2022 •

edited

Loading

ahrtr commented Jun 23, 2022 •

edited

Loading

nolouch commented Jun 23, 2022 •

edited

Loading

nolouch commented Jun 23, 2022 •

edited

Loading