-
Notifications
You must be signed in to change notification settings - Fork 4.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
grpc: Wait until resources finish cleaning up in Stop() and GracefulStop() #6489
Commits on Jul 31, 2023
-
transport/http2server: wait in Close() until loopyWriter terminated
When Server.Stop() returns it is not guaranteed that the loopyWriter go-routine terminated. This can lead to a panic or a testing.(*common).logDepth() race condition in Go Tests because t.Log is used after or during the testcase terminates. This can happen when: - a GRPC server is started in a Go test, - the GRPC logs are forwarded to t.Log, - loopyWriter.run logs an error after server.Stop() and the Test method returns. grpc@v1.57.0/internal/leakcheck is unable to detect that the loopyWriter go-routine continues to run after server.Stop() because it waits up to 10s for go-routines to terminate after a test finishes. The loopyWriter returns much faster after Stop() returns. To make server.Stop() wait until loopyWriter terminated: - rename the existing writerDone field, which is only used in tests, to loopyWriterDone, the writerDone channel is closed when the loopyWriter go-routine exited - change http2server.Close to wait until loopyWriterDone is closed
Configuration menu - View commit details
-
Copy full SHA for ad96f3c - Browse repository at this point
Copy the full SHA ad96f3cView commit details
Commits on Aug 9, 2023
-
Revert "transport/http2server: wait in Close() until loopyWriter term…
…inated" This reverts commit ad96f3c.
Configuration menu - View commit details
-
Copy full SHA for 26fd669 - Browse repository at this point
Copy the full SHA 26fd669View commit details -
server: Wait in GracefulStop and Stop until loopyWriter goroutine exits
It was not guaranteed that when Stop() and GracefulStop() returned, the loopyWriter go-routines terminated. This can lead to a panic or a testing.(*common).logDepth() race condition in Go Tests because t.Log is used after or during the testcase terminates. This can happen when: - a GRPC server is started in a Go test, - the GRPC logs are forwarded to t.Log, - loopyWriter.run logs an error after server.Stop() and the Test method returns. grpc@v1.57.0/internal/leakcheck is unable to detect that the loopyWriter go-routine continues to run after server.Stop() because it waits up to 10s for go-routines to terminate after a test finishes. The loopyWriter returns much faster after Stop() returns. To make ensure Stop and GracefulStop only return after the loopyWriter go-routine terminated: - rename http2Server.writerDone to loopyWriterDone, the channel was only used in test and is closed when loopyWriter returns - wait in http2Server.HandleStreams() until loopyWriterDone was closed - wait in Server.Stop() until all connections were removed, this is already done in GracefulStop. To abort GracefulStop() when Stop() is called a local copy of Server.lis and Server.conns was created and both fields were set to nil and then s.csv.Broadcast() was called to wake up an eventually sleeping GracefulStop() execution. This does not work anymore when we need to wait for connections to be removed in Stop() we need to check the length of Server.conns. Instead a new abortGracefulStop atomicBool is introduced, that is set to true to indicate that a GracefulStop() invocation should be aborted when Stop() is run.
Configuration menu - View commit details
-
Copy full SHA for cb7cfa7 - Browse repository at this point
Copy the full SHA cb7cfa7View commit details
Commits on Aug 11, 2023
-
Configuration menu - View commit details
-
Copy full SHA for 7eb46c0 - Browse repository at this point
Copy the full SHA 7eb46c0View commit details -
Configuration menu - View commit details
-
Copy full SHA for 5c056bd - Browse repository at this point
Copy the full SHA 5c056bdView commit details
Commits on Aug 28, 2023
-
Configuration menu - View commit details
-
Copy full SHA for 60700e0 - Browse repository at this point
Copy the full SHA 60700e0View commit details -
Configuration menu - View commit details
-
Copy full SHA for f0250c2 - Browse repository at this point
Copy the full SHA f0250c2View commit details -
Configuration menu - View commit details
-
Copy full SHA for e73ad2c - Browse repository at this point
Copy the full SHA e73ad2cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 17fd57e - Browse repository at this point
Copy the full SHA 17fd57eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 62a55c3 - Browse repository at this point
Copy the full SHA 62a55c3View commit details -
server: stopServerWorkers also in GracefulStop
The call was only done in Stop() and missing in GracefulStop()
Configuration menu - View commit details
-
Copy full SHA for 6143cf7 - Browse repository at this point
Copy the full SHA 6143cf7View commit details -
server: consolidate shared Stop and GracefulStop code
Move code that is used in Stop and GracefulStop to helper functions instead of duplicating it. This makes it easier to compare Stop() and GracefulStop.
Configuration menu - View commit details
-
Copy full SHA for 8a4ebe0 - Browse repository at this point
Copy the full SHA 8a4ebe0View commit details
Commits on Sep 7, 2023
-
Configuration menu - View commit details
-
Copy full SHA for 8bba089 - Browse repository at this point
Copy the full SHA 8bba089View commit details
Commits on Oct 20, 2023
-
Configuration menu - View commit details
-
Copy full SHA for ecd6289 - Browse repository at this point
Copy the full SHA ecd6289View commit details -
Configuration menu - View commit details
-
Copy full SHA for c0b97bb - Browse repository at this point
Copy the full SHA c0b97bbView commit details -
Revert "server: close listeners and wait for serveWG threads before d…
…raining" This reverts commit 0264c98ecd860431ac82a4e799cf6282e08e8a04.
Configuration menu - View commit details
-
Copy full SHA for b55bee0 - Browse repository at this point
Copy the full SHA b55bee0View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4b1e3eb - Browse repository at this point
Copy the full SHA 4b1e3ebView commit details
Commits on Oct 26, 2023
-
- drain transporters after waiting serve handlers to terminate - call closeServerTransports only on non graceful shutdowns, to abruptly close the connections
Configuration menu - View commit details
-
Copy full SHA for acc0243 - Browse repository at this point
Copy the full SHA acc0243View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8a58769 - Browse repository at this point
Copy the full SHA 8a58769View commit details -
tests: add server.Stop() and server.GracefulStop testcases
Add 2 testcases that ensure that: - GracefulStop blocks and waits until ongoing client grpc calls finis, - Stop() returns despite ongoing blocking grpc calls and clients receive a connection error
Configuration menu - View commit details
-
Copy full SHA for e85749a - Browse repository at this point
Copy the full SHA e85749aView commit details -
Delete internal/transport/http2_server.go.orig
The file was added accidentally
Configuration menu - View commit details
-
Copy full SHA for 11254c2 - Browse repository at this point
Copy the full SHA 11254c2View commit details -
Configuration menu - View commit details
-
Copy full SHA for ecfd5d1 - Browse repository at this point
Copy the full SHA ecfd5d1View commit details
Commits on Oct 27, 2023
-
Configuration menu - View commit details
-
Copy full SHA for 3feef89 - Browse repository at this point
Copy the full SHA 3feef89View commit details -
Configuration menu - View commit details
-
Copy full SHA for c03201f - Browse repository at this point
Copy the full SHA c03201fView commit details -
Configuration menu - View commit details
-
Copy full SHA for e7b4971 - Browse repository at this point
Copy the full SHA e7b4971View commit details
Commits on Oct 30, 2023
-
Configuration menu - View commit details
-
Copy full SHA for cb549f2 - Browse repository at this point
Copy the full SHA cb549f2View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8354300 - Browse repository at this point
Copy the full SHA 8354300View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8bc0f7b - Browse repository at this point
Copy the full SHA 8bc0f7bView commit details -
Configuration menu - View commit details
-
Copy full SHA for d976997 - Browse repository at this point
Copy the full SHA d976997View commit details -
Configuration menu - View commit details
-
Copy full SHA for 702ac45 - Browse repository at this point
Copy the full SHA 702ac45View commit details
Commits on Oct 31, 2023
-
Configuration menu - View commit details
-
Copy full SHA for 66b367c - Browse repository at this point
Copy the full SHA 66b367cView commit details