Possible deadlock with bidirectional streaming

[exelsior]

Hello.
Got interesting problem with bidirectional streaming through grpc connection in .Net.
I have three servers, like client <-> client-server <-> server.
client write in request stream requests, client-server process it (sort of), and server makes another requests to third party services (which is kinda slow). Problem occures when i write to request stream on client several dozens big requests (about 35-37 kb), server (through client-server) receives it, and starts process it. I think i should say that producer (client) is much faster than consumer (server).
Server relies on await foreach IAsyncEnumerable, so it reads first message, process it, write response to response stream and then read next request. But after processing first request, he can not write response to response stream, and consequently it can not read another request. I believe it's because of GRPC_ARG_HTTP2_WRITE_BUFFER_SIZE parameter which is 65k by default. And if channel buffer is full of requests, there is literally no space for response, which is, basically, a deadlock. The most obvious solution - read all from request channel, store this request in some array and then just foreach it. But, unfortunately, this project has a use case, when server uses response data as a request for another service, and response is, basically, this data with some enrichments.
So with this case in mind, i can not use obvious solution, because in that case i would store all response in memory. And it can be hundreds of megabytes.
I tried to increase InitialHttp2StreamWindowSize in SocketHttpHandler which i use for GrpcChannel creation, to it maximum and it resolves the problem, but i realize it is a temporary solution. Setting EnableMultipleHttp2Connections in the same SocketHttpHandler doesn't help at all.
I created MRE for ensuring that this is not our code peculiarity
Is there some more permanent solution? Or i just do smth wrong?

I'm using Grpc.AspNetCore 2.63.0
OS: Macos Sonoma 14.1.1
Device: Macbook pro m1
Dotnet: 8.0.300

MRE: https://drive.google.com/file/d/13JFrQp8KeSNZ6D8TQJjpKl5rNbaBYB7s/view?usp=drive_link

[JamesNK]

GRPC_ARG_HTTP2_WRITE_BUFFER_SIZE is used by Grpc.Core. It isn't a Grpc.AspNetCore or Grpc.Net.Client setting.

HTTP/2 flow control request and response use different buffers. A server that isn't reading request data fast enough shouldn't impact its ability to send response data.

https://medium.com/coderscorner/http-2-flow-control-77e54f7fd518

[exelsior]

Hello @JamesNK.
Thank you for your answer. Different buffer - definitely good news.
One less thing.
I tried to reduce message / increase InitialHttp2StreamWindowSize and it works.
But with the initial windowsize and relatively big message - it doesn't.
Do you have any ideas why server can not send his response?

[JamesNK]

The server might not be able to send the complete response because the buffer fills up, so it's waiting for the client to read it.

[exelsior]

@JamesNK Yes, that's definitely the case.
But i got interesting detail. If server reads request stream till the end, stores all requests in memory and then starts to process it, sending response to ResponseStream, it works perfectly. Seems like buffer becomes empty after reading requests, so we can write there response without any problems.
I read article about http2 flow control you provided (thanks btw), but are you quite sure there is different buffers for request stream and response stream?
Just strange behavior and i can not understand why it's stuck.

[exelsior]

I've ran out of ideas.
Seems like backpressure doesn't working.
https://github.com/exelsior/grpc-deadlock/tree/main

[exelsior]

Well. actually no. It's working. But because of full buffer, there is no update window and the whole system goes deadlock. Is it correct behavior?