Add connection pool DEBUG logs & tracing.

[tomchristie]

Escalated from discussion #505

^{Originally posted by inikolaev February 7, 2022}
Hello!

Can you advise some way to monitor connection pool, specifically for the AsyncClient?

What I would like to know is the following:

• How many request are awaiting a connection
• How many connections are being used right now (in-flight requests)
• Some statistics on connection open/close

I could potentially get some numbers using existing request and response hook, what I don't know is when request hook is fired - does it fire before connection is acquired or after - it's not very clear to me from the documentation. But this should be easy to check.

Perhaps there could be added hooks into the connection pool or maybe someone from the community does this in a different way somehow already?

Currently the "trace" extension doesn't provide any visibility around the connection pooling.

It would be neat if we had events around this. I'm not sure exactly what set of events we'd want, but essentially we'd like to be able to track information around when connections are added or removed from the pool, and when connections become active or idle.

The first step here would be to determine what set of events we'd want to see, sketch out how the documentation ought to look once we've added those. Once that's done we can tackle the implementation.

[inikolaev]

Regarding the events I was thinking about these:

• pool.connection.wait - waiting for a connection from the pool, but maybe here we need several events, since want might fail due to timeout - similar to what Trace does already
  • pool.connection.wait.started
  • pool.connection.wait.complete
  • pool.connection.wait.failed
• pool.connection.created - a new connection added to the pool
• pool.connection.acquired - connection retrieved from the pool
• pool.connection.released - connection returned into the pool
• pool.connection.closed - connection closed and removed from the pool

Do you think these make sense? It would at least cover my needs, but maybe there are some metrics I'm not thinking about.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.