Optimize connection pool implementation #924

MarkusSintonen · 2024-06-10T18:39:46Z

Summary

Second PR in series of optimizations to improve performance of httpcore and even reach performance levels of aiohttp (and urllib3 library).
Related discussion encode/httpx#3215 (comment)

Optimizes the connection pool by reducing the time complexity of the idling connections checks. Also it no longer checks the sockets readable status on every pool operation. Pools has_expired check uses socket polling via is_readable check which is relatively expensive. So now the polling is done using smudged intervals (smudging to avoid all polls being done at the exactly same time). This still should have relatively low chance of encountering broken keep alive connections when connection is picked up from the pool.

Async previously:

Async with PR:

Async request latency is not so stable yet (as this doesn't include #922) but the overall duration of the benchmark improves by 7.5x. (The difference diminishes when server latency increases over 100ms or so.)

Sync previously:

Sync with PR:

Sync request latency improves by x2.5. With this httpcore has same performance as urllib3 library.

Checklist

I understand that this PR may be closed in case there was no previous discussion. (This doesn't apply to typos!)
I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
I've updated the documentation accordingly.

karpetrosyan · 2024-06-11T19:41:14Z

httpcore/_async/connection_pool.py

@@ -238,24 +238,27 @@ def _assign_requests_to_connections(self) -> List[AsyncConnectionInterface]:
        those connections to be handled seperately.
        """
        closing_connections = []
+        idling_connections = {c for c in self._connections if c.is_idle()}


It seems like we use this collection just for tracking the count of idle connections. Maybe it should just be an integer for simplicity?

~~With integer we would need to recheck all the connections again. We can not just decrement below in the if-elif branches as we could end up going negative etc~~

Heh sorry, you are right. We can just check eg the expired one if it was also idle one and decrement!

Does that part significantly change the performance? Did you run any tests? Maybe we are gaining the same performance boost by only using polling?

When I checked with pyinstrument it showed this exact spot as a hot spot spending time iterating the connections in outer and inner loops. Ill rerun the benchmark to check it again

Anyways I pushed now the integer fix

Without the fix in _assign_requests_to_connections the performance starts to deteriorate when the amount of connections in the pool increases (which is not surprising as there is the loop in the loop).

With 20 concurrency (sync):

With 40 concurrency (sync):

karpetrosyan · 2024-06-11T20:14:01Z

I have similar results without this PR. Am I doing something wrong?
I am using benchmarks script from your another PR

Here is what I got on my machine without this PR:

UPDATE:

However, the test results for httpcore vary between 3500 and 5500 for me (without this change).

MarkusSintonen · 2024-06-12T04:15:12Z

I have similar results without this PR. Am I doing something wrong?

What kind of system are you on and python version?

httpcore/_async/connection_pool.py

T-256 · 2024-06-12T17:23:44Z

httpcore/_async/http11.py

+        # Checking the readable status is relatively expensive so check it at a lower frequency.
+        if (now - self._network_stream_used_at) > self._socket_poll_interval():
+            self._network_stream_used_at = now
+            server_disconnected = (
+                self._state == HTTPConnectionState.IDLE
+                and self._network_stream.get_extra_info("is_readable")
+            )
+            if server_disconnected:
+                return True
+
+        return False


Actually, I'm not happy with interval calculating.

Could we improve is_readable instead? FWIW I noticed anyio and sync backend use httpcore._utils.is_socket_readable while trio backend uses its own, @MarkusSintonen is there any benchmark difference when you switch backend to trio?

For improving is_readable in get_extra_info:

Always assume it is readable and turn it to false by specified events such as received close socket from server.

Use synchronized Event on readability status change.

I didn't go deep to search these cases are possible or not. They are only my opinions 🤷‍♂️

Trio also suffers badly from the constantly happening socket polling.

Without intervalled socket polling:

With intervalled socket polling:

So in trio its over 5x slower when constantly doing the socket polling.

For improving is_readable in get_extra_info:
Always assume it is readable and turn it to false by specified events such as received close socket from server.

Im not aware of anyway to to get events about socket getting closed. As far as I know the only way to know it is to use the socket. 🤔 But I agree the is_readable could be better so its not so heavy weight. We could make it just a flag based so in networking side we just set some boolean flag when we detect a network error on usage. This has a downside that we have greater probability of giving out already broken connections from the pool. But as far as I know this is how its usually done.

Other options would be to move the intervalled polling into the network backends side. Or get even more elaborate and run the socket polling in a specific interval via loop.call_later which run the actual poll via loop.run_in_executor to avoid any possible nonasync socket IO in the async land. Gets easily hairy 😄

T-256

Except that interval mechanism (perhaps could be excluded from this PR), it's LGTM, Thank You!

MarkusSintonen · 2024-06-12T18:14:00Z

(perhaps could be excluded from this PR)

That was the beef of the PR as the constantly happening socket polling is the most expensive thing in the whole pooling implementation 😅 (loop complexity gets shadowed by the socket polling)

tomchristie · 2024-06-13T08:44:29Z

Let's get this split into two seperate PRs please, so we can look at each in isolation and determine if it's a benefit.

MarkusSintonen · 2024-06-13T15:47:14Z

Let's get this split into two seperate PRs please, so we can look at each in isolation and determine if it's a benefit.

Will do, FYI here was the pool without the loop complexity fix #924 (comment)

MarkusSintonen · 2024-06-13T18:38:11Z

@tomchristie / @T-256
I have now split the PR into two. Ill close this one.

This was referenced Jun 10, 2024

Improve async performance. encode/httpx#3215

Open

Add benchmarking script #923

Merged

MarkusSintonen force-pushed the optimize-conn-pool branch 2 times, most recently from 6819e1c to 1358872 Compare June 11, 2024 19:39

karpetrosyan reviewed Jun 11, 2024

View reviewed changes

T-256 reviewed Jun 12, 2024

View reviewed changes

httpcore/_async/connection_pool.py Outdated Show resolved Hide resolved

T-256 reviewed Jun 12, 2024

View reviewed changes

T-256 approved these changes Jun 12, 2024

View reviewed changes

MarkusSintonen force-pushed the optimize-conn-pool branch from bb3cbe1 to 0694653 Compare June 13, 2024 17:43

Optimize connection pool

b6b119c

MarkusSintonen force-pushed the optimize-conn-pool branch from 0694653 to b6b119c Compare June 13, 2024 17:43

MarkusSintonen marked this pull request as draft June 13, 2024 17:50

This was referenced Jun 13, 2024

Connection pool optimization: move socket polling from expiry checks to connection usage #928

Open

Connection pool optimization: reduce connection maintenance loops complexity #929

Open

MarkusSintonen closed this Jun 13, 2024

MarkusSintonen deleted the optimize-conn-pool branch June 13, 2024 18:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Optimize connection pool implementation #924

Optimize connection pool implementation #924

MarkusSintonen commented Jun 10, 2024 •

edited

Loading

karpetrosyan Jun 11, 2024

MarkusSintonen Jun 11, 2024 •

edited

Loading

MarkusSintonen Jun 11, 2024

karpetrosyan Jun 11, 2024

MarkusSintonen Jun 11, 2024

MarkusSintonen Jun 11, 2024

MarkusSintonen Jun 12, 2024

karpetrosyan commented Jun 11, 2024 •

edited

Loading

MarkusSintonen commented Jun 12, 2024

T-256 Jun 12, 2024

MarkusSintonen Jun 12, 2024

MarkusSintonen Jun 12, 2024 •

edited

Loading

MarkusSintonen Jun 12, 2024

T-256 left a comment •

edited

Loading

MarkusSintonen commented Jun 12, 2024 •

edited

Loading

tomchristie commented Jun 13, 2024

MarkusSintonen commented Jun 13, 2024 •

edited

Loading

MarkusSintonen commented Jun 13, 2024

Optimize connection pool implementation #924

Optimize connection pool implementation #924

Conversation

MarkusSintonen commented Jun 10, 2024 • edited Loading

Summary

Checklist

Choose a reason for hiding this comment

MarkusSintonen Jun 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

karpetrosyan commented Jun 11, 2024 • edited Loading

MarkusSintonen commented Jun 12, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MarkusSintonen Jun 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

T-256 left a comment • edited Loading

Choose a reason for hiding this comment

MarkusSintonen commented Jun 12, 2024 • edited Loading

tomchristie commented Jun 13, 2024

MarkusSintonen commented Jun 13, 2024 • edited Loading

MarkusSintonen commented Jun 13, 2024

MarkusSintonen commented Jun 10, 2024 •

edited

Loading

MarkusSintonen Jun 11, 2024 •

edited

Loading

karpetrosyan commented Jun 11, 2024 •

edited

Loading

MarkusSintonen Jun 12, 2024 •

edited

Loading

T-256 left a comment •

edited

Loading

MarkusSintonen commented Jun 12, 2024 •

edited

Loading

MarkusSintonen commented Jun 13, 2024 •

edited

Loading