3.11.5 regression: StreamWriter.__del__ fails if event loop is already closed

[bmerry]

Bug report

Bug description:

PR #107650 added a StreamWriter.__del__ that emits a ResourceWarning if a StreamWriter is not closed by the time it is garbage collected, and it has been backported as 3.11.5. However, if the event loop has already been closed by the time this happens, it causes a RuntimeError message to be displayed. It's non-fatal because exceptions raised by __del__ are ignored, but it causes an error message to be displayed to the user (as opposed to ResourceWarning, which is only shown when one opts in).

Code like the following used to run without any visible error, but now prints a traceback (and does not report the ResourceWarning for the unclosed StreamWriter when run with -X dev):

#!/usr/bin/env python3

import asyncio

async def main():
    global writer
    reader, writer = await asyncio.open_connection("127.0.0.1", 22)

asyncio.run(main())

Output in 3.11.5:

Exception ignored in: <function StreamWriter.__del__ at 0x7fd54ee11080>
Traceback (most recent call last):
  File "/usr/lib/python3.11/asyncio/streams.py", line 396, in __del__
  File "/usr/lib/python3.11/asyncio/streams.py", line 344, in close
  File "/usr/lib/python3.11/asyncio/selector_events.py", line 860, in close
  File "/usr/lib/python3.11/asyncio/base_events.py", line 761, in call_soon
  File "/usr/lib/python3.11/asyncio/base_events.py", line 519, in _check_closed
RuntimeError: Event loop is closed

CPython versions tested on:

3.11

Operating systems tested on:

Linux

Linked PRs

• gh-109538: Catch closed loop runtime error and issue warning #111983
• [3.11] gh-109538: Catch closed loop runtime error and issue warning (GH-111983) #112141
• [3.12] gh-109538: Catch closed loop runtime error and issue warning (GH-111983) #112142

[dpr-0]

Same problem here

[smirnoffs]

I have the same problem with 3.11.6. Thanks for reporting it.

[pfps]

Is there a workaround?

[dpr-0]

Is there a workaround?

Use gc.collect() may help

Here is my workaround.

@pytest.fixture
def event_loop():
    policy = asyncio.get_event_loop_policy()
    loop = policy.new_event_loop()
    loop.set_debug(True)
    yield loop
    gc.collect()
    loop.close()

[pfps]

@dpr-0 Thanks, but just adding that to my program didn't prevent the messages.

[dpr-0]

@dpr-0 Thanks, but just adding that to my program didn't prevent the messages.

Hope you can find the problem T^T

[dtatarkin]

Same problem for python 3.12.0

[gvanrossum]

Sorry for the inconvenience. Can you (or anyone else reading this) submit a PR for the main branch? We will then backport the PR to 3.12 and 3.11.

[bmerry]

Sorry for the inconvenience. Can you (or anyone else reading this) submit a PR for the main branch? We will then backport the PR to 3.12 and 3.11.

What is the desired behaviour if the event loop is already closed when the StreamWriter is destroyed? Should __del__ just become a no-op?

[gvanrossum]

What you expected -- I presume it'd still issue the resource warning, just not the "loop is closed" message.

[dpr-0]

@gvanrossum I would imagine like this

def __del__(self, warnings=warnings):
    if not self._transport.is_closing():
        try:
            self.close()
        except RuntimeError:
            warnings.warn(f"loop is closed", ResourceWarning)
        else:
            warnings.warn(f"unclosed {self!r}", ResourceWarning)

If this one ok, I can submit a PR for this issues

[gvanrossum]

Sure, let’s give that a try as a PR.

Maybe separately it would be nice to gc.collect() in loop.close(). That’s a larger discussion though.

[RoTorEx]

Did anyone find a solution?

[gvanrossum]

Did anyone find a solution?

It's fixed in the main, 3.12 and 3.11 branches. It will be fixed in the next official releases of 3.11 and 3.12 from python.org (within 1-2 months).

[LostInDarkMath]

Did anyone find a solution?

It's fixed in the main, 3.12 and 3.11 branches. It will be fixed in the next official releases of 3.11 and 3.12 from python.org (within 1-2 months).

Is this fix included in 3.11.7?
Because I'm still able to reproduce it with Python 3.11.7

Edit: nevermind, I used 3.11.6 by accident. All good

[mdczaplicki]

Just so you know, on python 3.11.9 with this setting for pytest:

[tool.pytest.ini_options]
asyncio_mode = "auto"
filterwarnings = ["error"]

the suite still raises an error from ignored warning:

    def _warn_teardown_exception(
        hook_name: str, hook_impl: HookImpl, e: BaseException
    ) -> None:
        msg = "A plugin raised an exception during an old-style hookwrapper teardown.\n"
        msg += f"Plugin: {hook_impl.plugin_name}, Hook: {hook_name}\n"
        msg += f"{type(e).__name__}: {e}\n"
        msg += "For more information see https://pluggy.readthedocs.io/en/stable/api_reference.html#pluggy.PluggyTeardownRaisedWarning"  # noqa: E501
>       warnings.warn(PluggyTeardownRaisedWarning(msg), stacklevel=5)
E       pluggy.PluggyTeardownRaisedWarning: A plugin raised an exception during an old-style hookwrapper teardown.
E       Plugin: unraisableexception, Hook: pytest_runtest_setup
E       PytestUnraisableExceptionWarning: Exception ignored in: <function StreamWriter.__del__ at 0xffffb9f4d620>
E       
E       Traceback (most recent call last):
E         File "/usr/local/lib/python3.11/asyncio/streams.py", line 411, in __del__
E           warnings.warn("loop is closed", ResourceWarning)
E       ResourceWarning: loop is closed
E       
E       For more information see https://pluggy.readthedocs.io/en/stable/api_reference.html#pluggy.PluggyTeardownRaisedWarning

../virtualenvs/project-bTydf30B-py3.11/lib/python3.11/site-packages/pluggy/_callers.py:50: PluggyTeardownRaisedWarning

    def unraisable_exception_runtest_hook() -> Generator[None, None, None]:
        with catch_unraisable_exception() as cm:
            yield
            if cm.unraisable:
                if cm.unraisable.err_msg is not None:
                    err_msg = cm.unraisable.err_msg
                else:
                    err_msg = "Exception ignored in"
                msg = f"{err_msg}: {cm.unraisable.object!r}\n\n"
                msg += "".join(
                    traceback.format_exception(
                        cm.unraisable.exc_type,
                        cm.unraisable.exc_value,
                        cm.unraisable.exc_traceback,
                    )
                )
>               warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))
E               pytest.PytestUnraisableExceptionWarning: Exception ignored in: <function StreamWriter.__del__ at 0xffff878ed620>
E               
E               Traceback (most recent call last):
E                 File "/usr/local/lib/python3.11/asyncio/streams.py", line 411, in __del__
E                   warnings.warn("loop is closed", ResourceWarning)
E               ResourceWarning: loop is closed

../virtualenvs/project-bTydf30B-py3.11/lib/python3.11/site-packages/_pytest/unraisableexception.py:78: PytestUnraisableExceptionWarning

Is it expected? Should I add those warnings as ignored to my pytest setup?

[gvanrossum]

@mdczaplicki That error might be due to pytest -- I cannot tell from the output you're posting. If you really think this is still not fixed in CPython 3.11.9, can you open a new issue with complete instructions for reproducing the problem? (Ideally also testing in 3.12.)