Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GRPC Service (Server Side) not causing exception if unable to send data to client #334

Open
Uight opened this issue Sep 9, 2024 · 0 comments

Comments

@Uight
Copy link

Uight commented Sep 9, 2024

Im having a problem with server to client communication. I have a Server implementing the following interface method:

[Service]
public interface ICommandApiService
{
IAsyncEnumerable SubscribeForUserDialogEvents(CallContext context = default);
}

    public async IAsyncEnumerable<UserDialogEventData> SubscribeForUserDialogEvents(CallContext context = default)
    {
        var cancel = context.CancellationToken;
        var clientSubscription = new ClientUserDialogSubscription();
        clientSubscriptions.Add(clientSubscription);
        
        logger.Info($"A subscriber for user dialog events connected. Current subscriber count {clientSubscriptions.Count}");
        var delayCancellationTokenSource = CancellationTokenSource.CreateLinkedTokenSource(cancel);
        var waitForCancellationTask = Task.Delay(Timeout.Infinite, delayCancellationTokenSource.Token);

        try
        {
            // Immediately push a success message so the client can verify the connection is up and running.
            yield return new UserDialogEventData
            {
                EventType = UserDialogEventType.ConfirmSuccess
            };

            //Push messages for all currently opened dialogs
            foreach (var openDialog in currentlyOpenedDialogs)
            {
                yield return openDialog.Value;
            }

            var nextTcs = clientSubscription.UserDialogEventRequestBatch.NextBatch;

            while (!cancel.IsCancellationRequested)
            {
                var notificationTask = nextTcs.Task;
                var heartbeatTaskDelay = Task.Delay(2500, cancel);
                var completedTask = await Task.WhenAny(notificationTask, heartbeatTaskDelay, waitForCancellationTask);

                if (cancel.IsCancellationRequested)
                {
                    break;
                }

                if (completedTask == heartbeatTaskDelay)
                {
                    yield return new UserDialogEventData
                    {
                        EventType = UserDialogEventType.KeepAlive
                    };
                    continue;
                }

                var notificationBatch = await nextTcs.Task;
                nextTcs = notificationBatch.NextBatch;

                //Send new event to client
                yield return notificationBatch.UserDialogEventRequest;

                // Notify successfully send
                clientSubscription.AcknowledgmentTcs.SetResult(true);
            }
        }
        finally
        {
            clientSubscriptions.Remove(clientSubscription);
            clientSubscription.AcknowledgmentTcs.SetResult(false);
            logger.Info("A subscriber for user dialog events disconnected");
            
            await delayCancellationTokenSource.CancelAsync();
            waitForCancellationTask.Wait(TimeSpan.FromMilliseconds(100));
            delayCancellationTokenSource.Dispose();
        }
    }

this works fine for the most part and also when the client cleanly disconnects this works fine. This sends out new Data when triggered from another method and stops when the cancellation is requested. But we identified one problem here which is why we added the Keepalive messages all 2,5 seconds but that doesnt help.

The problem is that in the production environment our software runs in sometimes the connection is lost. (Most easily reproduced by just disconnecting the LAN Cable between client and server. In this case i would expect that in end up failing at on of the yield returns. However the yield returns just keep working and the loop things its still sending our data even after 10 Minutes it doesn't fail.
The client side pretty much immediately recognizes the connection loss but the server never does.
The most problematic for us is that we tried to monitor the send options to check if every connected client did receive the message. But in case of a connection loss we still think we can send out without an error but that isn't the case.

Obviosly there would be some Solutions here like using a two way stream with acknowledgement and timeout or similar things. But i would still like to know why the server doesnt recognize the connection loss. Is the code wrong? or is the server setup badly?

Server setup:

        var builder = WebApplication.CreateBuilder();
        var services = builder.Services;
        
        services.AddSingleton(typeof(CommandApiService), commandService);
        services.AddGrpc(options =>
        {
            const int fiftyMegaBytes = 50 * 1024 * 1024;
            options.MaxReceiveMessageSize = fiftyMegaBytes;
            options.MaxSendMessageSize = fiftyMegaBytes;
            options.EnableDetailedErrors = true;
        });
        
        services.AddCodeFirstGrpc(config => { config.ResponseCompressionLevel = System.IO.Compression.CompressionLevel.Optimal; });
        
        builder.WebHost.ConfigureKestrel(options =>
        {
            options.ListenAnyIP(port, listenOptions => listenOptions.Protocols = HttpProtocols.Http2);
            options.Limits.MaxRequestBodySize = 50 * 1024 * 1024;
        });
        
        app = builder.Build();
        app.UseRouting();
        
        app.MapGrpcService<CommandApiService>(); 
        
        app.RunAsync();
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant