RuntimeError: Event loop is closed / unable to perform operation on <TCPTransport closed=True reading=False>; the handler is closed

[eigenein]

Hi,

I'm splitting the issue from here: encode/uvicorn#1251.

Environment

• Python 3.10.2
• httpcore==0.14.7

Example

import asyncio
from time import sleep

from httpcore._async import AsyncConnectionPool

pool = AsyncConnectionPool(keepalive_expiry=1.0)


async def main():
    response = await pool.request("GET", "https://httpbin.org/get")
    assert response.status == 200


# First request creates a new connection within the event loop.
asyncio.run(main())

# Sleep for longer than the `keepalive_expiry` to make the pool close expired connections.
sleep(2.0)

# Sending another request from a different event loop.
# Why different? For me that's the case in an ASGI application (Gunicorn + Django),
# where ingoing requests may be handled in different event loops.
# I'm trying to keep a global connection pool in the app which results in calling
# `pool.request` from different event loops.
asyncio.run(main())

Expected

I assume that httpcore should silently ignore the error since we want the connection closed and it just isn't valid anymore.

Actual

Traceback (most recent call last):
  File "/Users/eigenein/GitHub/blitz-lab/src/mre.py", line 16, in <module>
    asyncio.run(main())
  File "/usr/local/Cellar/python@3.10/3.10.2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/Cellar/python@3.10/3.10.2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 641, in run_until_complete
    return future.result()
  File "/Users/eigenein/GitHub/blitz-lab/src/mre.py", line 10, in main
    response = await pool.request("GET", "https://httpbin.org/get")
  File "/Users/eigenein/GitHub/blitz-lab/venv/lib/python3.10/site-packages/httpcore/_async/interfaces.py", line 41, in request
    response = await self.handle_async_request(request)
  File "/Users/eigenein/GitHub/blitz-lab/venv/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 220, in handle_async_request
    await self._close_expired_connections()
  File "/Users/eigenein/GitHub/blitz-lab/venv/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 188, in _close_expired_connections
    await connection.aclose()
  File "/Users/eigenein/GitHub/blitz-lab/venv/lib/python3.10/site-packages/httpcore/_async/connection.py", line 159, in aclose
    await self._connection.aclose()
  File "/Users/eigenein/GitHub/blitz-lab/venv/lib/python3.10/site-packages/httpcore/_async/http11.py", line 212, in aclose
    await self._network_stream.aclose()
  File "/Users/eigenein/GitHub/blitz-lab/venv/lib/python3.10/site-packages/httpcore/backends/asyncio.py", line 48, in aclose
    await self._stream.aclose()
  File "/Users/eigenein/GitHub/blitz-lab/venv/lib/python3.10/site-packages/anyio/streams/tls.py", line 168, in aclose
    await self.transport_stream.aclose()
  File "/Users/eigenein/GitHub/blitz-lab/venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 1163, in aclose
    self._transport.close()
  File "/usr/local/Cellar/python@3.10/3.10.2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/selector_events.py", line 700, in close
    self._loop.call_soon(self._call_connection_lost, None)
  File "/usr/local/Cellar/python@3.10/3.10.2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 745, in call_soon
    self._check_closed()
  File "/usr/local/Cellar/python@3.10/3.10.2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 510, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

With uvloop you get something like (quoting the differing part at the bottom):

  File "anyio/streams/tls.py", line 168, in aclose
    await self.transport_stream.aclose()
  File "anyio/_backends/_asyncio.py", line 1159, in aclose
    self._transport.write_eof()
  File "uvloop/handles/stream.pyx", line 696, in uvloop.loop.UVStream.write_eof
    self._ensure_alive()
  File "uvloop/handles/handle.pyx", line 159, in uvloop.loop.UVHandle._ensure_alive
    raise RuntimeError(
RuntimeError: unable to perform operation on <TCPTransport closed=True reading=False 0x55ee6d824e80>; the handler is closed

[tomchristie]

👀 - I really wouldn't know what behaviour to expect with that usage.

I'd strongly recommend that you instantiate the pool within the async context, rather than outside it...

async def main():
    async with AsyncConnectionPool(keepalive_expiry=1.0) as pool:
        response = await pool.request("GET", "https://httpbin.org/get")
        assert response.status == 200

Why different? For me that's the case in an ASGI application (Gunicorn + Django),
where ingoing requests may be handled in different event loops.

Right. I don't know enough about Django under ASGI to figure out if there's some top-level async context where you can instantiate an httpx connection pool, that'll be properly shared. Might be something to figure out on one of their discussion groups.

[provinzkraut]

Coming from encode/httpx/issues/2058, I feel like expiring connections from a pool when their associated event loop has been closed would be the natural and logical thing to do. As I've shown in encode/httpx/issues/2058#issuecomment-1260669184, the current behavior makes AsyncConnectionPool not thread safe or "loop safe". I think it's a wrong assumption to make that there will only ever be one event loop during the runtime of an application, as this is neither guaranteed nor expected.

It also wouldn't be too complicated to deal with this, as the following proof of concept shows:

import asyncio
from httpcore._async.connection_pool import AsyncConnectionPool

class LoopSafeConnectionPool(AsyncConnectionPool):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self._connection_event_loops = {}

    async def _attempt_to_acquire_connection(self, status):
        res = await super()._attempt_to_acquire_connection(status)
        if status.connection:
            self._connection_event_loops[status.connection] = asyncio.get_event_loop()
        return res

    async def _close_expired_connections(self):
        current_loop = asyncio.get_event_loop()
        for idx, connection in reversed(list(enumerate(self._pool))):
            connection_loop = self._connection_event_loops[connection]
            if connection_loop.is_closed() or current_loop is not connection_loop:
                self._pool.pop(idx)

        await super()._close_expired_connections()

[abersheeran]

I'm having the same problem, do we have a plan to fix it?

[tomchristie]

We don't want to silently ignore this, or else the user is instantiating a shared connection pool without any awareness that the connections are being closed.

Options:

• We can raise an explicit exception here, noting that this usage is not supported. 🤨
• We can add a note in the docs. 🧐
• We can explicitly enforce that async clients can only be instantiated using the context-managed style. 😎

(Interesting that trio does not raise an exception here.)