grpc connection reset logs when etcdctl get <key>

[niharkawli]

Version:
3.4.18
Issue:
etcd with zap logger logs grpc stream failure logs in warn level every time we run any etcdctl get command.

{"level":"warn","ts":"2022-01-18T11:45:24.564+0530","caller":"grpclog/grpclog.go:60","msg":"transport: http2Server.HandleStreams failed to read frame: read tcp 127.0.0.1:2379->127.0.0.1:58686: read: connection reset by peer"}
{"level":"warn","ts":"2022-01-18T11:45:25.200+0530","caller":"grpclog/grpclog.go:60","msg":"transport: http2Server.HandleStreams failed to read frame: read tcp 127.0.0.1:2379->127.0.0.1:58691: read: connection reset by peer"}
{"level":"warn","ts":"2022-01-18T11:45:25.528+0530","caller":"grpclog/grpclog.go:60","msg":"transport: http2Server.HandleStreams failed to read frame: read tcp 127.0.0.1:2379->127.0.0.1:58694: read: connection reset by peer"}
{"level":"warn","ts":"2022-01-18T11:45:25.872+0530","caller":"grpclog/grpclog.go:60","msg":"transport: http2Server.HandleStreams failed to read frame: read tcp 127.0.0.1:2379->127.0.0.1:58698: read: connection reset by peer"}

This issue is still seen in the latest 3.4.x branches.
Similar logs are seen frequently in all the members in our cluster and it becomes difficult to catch actual warning messages in the log because of these messages.

Fix PR:
#12406
Need to be backported to 3.4.x branch and a release created (possibly 3.4.19).

Reproduction step:
Can be reproduced on standalone etcd as well.

1. Run standalone etcd with zap logger.
   etcd --logger=zap
2. Run etcdctl get command 5 times.
   for i in {1..5}; do etcdctl get a; done
3. Check etcd logs. We see warn logs like the ones shown above.

The issue seems to be fixed in v3.5.1. However, we have an issue with migrating to 3.5.x version immediately because the in-service downgrade from 3.5 to 3.4 is not supported. Hence, it would be good to have the fix backported to v3.4.x branches.

[kkkkun]

/assign

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.