Fix network stack stability issues #10080
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
While analyzing a possible memory leak, we encountered several coroutine exception messages, which unfortunately do not provide any information about what exactly went wrong, as exception diagnostics were previously only logged at the notice level.
Although there is locking involved here, it shoudln't take too long for the thread to actually acquire it, since there aren't that many threads dealing with endpoint clients concurrently. It's just wasting pointless time trying to obtain a CPU slot.
Closing and re-opening that very same log file shouldn't reset the counter, otherwise some log files may exceed the max limit per file as their offset indicator is reset each time they are re-opened.
…ging On incoming connection timeout we log the remote endpoint which isn't available if it was already disconnected - an exception is thrown. Get it as long as we're still connected not to lose it, nor to get an exception.
When a HTTP connection dies prematurely while the response is sent, `http::async_write()` sets the error code to something like broken pipe for example. When calling `async_flush()` afterwards, it sometimes happens that this never returns. This results in a resource leak as the coroutine isn't cleaned up. This commit makes the individual functions throw exceptions instead of silently ignoring the errors, resulting in the function terminating early and also resulting in an error being logged as well.
icinga-probot
bot
added
area/api
When the `Desconnect()` method is called, clients are not disconnected immediately. Instead, a new coroutine is spawned using the same strand as the other coroutines. This coroutine calls `async_shutdown` on the TCP socket, which might be blocking. However, in order not to block indefintely, the `Timeout` class cancels all operations on the socket after `10` seconds. Though, the timeout does not trigger the handler immediately; it creates spawns another coroutine using the same strand as in the `JsonRpcConnection` class. This can cause unexpected delays if e.g. `HandleIncomingMessages` gets resumed before the coroutine from the timeout class. Apart from that, the coroutine for writing messages uses the same condition, making the two symmetrical.
…global thread pool
yhabteab
approved these changes
Nov 14, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport of
CpuBoundWorkusage inJsonRpcConnection::Disconnect()#9992CpuBoundWorkusages inlib/remote#9994m_LogMessageCountwhen rotating #9999CpuBoundWorkusage #10140