Ability to configure internal logging level

[gao-artur]

Today, when the retry policy is configured to handle Unavailable status, for example, each retry will be logged as an error.

fail: Grpc.Net.Client.Internal.GrpcCall[6]
      Error starting gRPC call.
      Grpc.Core.RpcException: Status(StatusCode="Unavailable", Detail="Error connecting to subchannel.", DebugException="System.Net.Sockets.SocketException (10061): No connection could be made because the target machine actively refused it.
         at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
         at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
         at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|281_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
         at Grpc.Net.Client.Balancer.Internal.SocketConnectivitySubchannelTransport.TryConnectAsync(ConnectContext context)")
         at Grpc.Net.Client.Balancer.Internal.ConnectionManager.PickAsync(PickContext context, Boolean waitForReady, CancellationToken cancellationToken)
         at Grpc.Net.Client.Balancer.Internal.BalancerHttpHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
         at Grpc.Net.Client.Internal.GrpcCall`2.RunCall(HttpRequestMessage request, Nullable`1 timeout)

This creates undesirable noise in the error channel. If the connection succeeds after a few retries, I'd want to see all the retry failures as warnings. And since the library throws RpcException after the final failure, the application writer can decide what logging level is appropriate in each case.

The simple fix would be to change the _errorStartingCall level from Error to Warning.

The more flexible solution would allow users to override specific message levels similar to how it's implemented in EF Core.

Currently, we just suppress all messages from the Grpc.Net.Client.Internal.GrpcCall namespace, which is not the perfect solution as the client may look stuck if a long retry policy is used.

[JamesNK]

This should be changed.

Options:

1. Either lower the log level to info all the time.
2. Lower the log level only if the call will be retried.

[gao-artur]

I vote for option 1. With the option 2 the error will be logged twice: once by the library, then once again by the user code. The first log also not very useful for the user because it doesn't include failed service and method name and the stuck trace won't include the call path in the user app, hence it's imposible to understand which call failed.