Tests sometimes hang

[Marenz]

What happened?

Sometimes the tests hang. Here is a log of the problem: https://gist.github.com/Marenz/1ace8c7c0ccf01db70ceee8f767bb6f9#file-different-eventloop-py-L188

Notes:

• Thread is always the same when the problem appears
• Possibly related to channels / Timer?
• Works when using the old event_loop replacement method

What did you expect instead?

.

Affected version(s)

No response

Affected part(s)

Build script, CI, dependencies, etc. (part:tooling)

Extra information

.

[llucax]

This started when merging pull request #54 introduced a timing issue with tests, making them flaky in amd64 but probably consistently failing in arm64 because the CI runs on qemu, which is extremely sloooooowwww.

Possibly related to channels / Timer?
The channels dependency was bumped in the mentioned PR via the bump of client-dispatch.

The issue seems to be that some condition variable is run in a different loop than the one it was created:

    | RuntimeError: <asyncio.locks.Condition object at 0x7f3d2bac0e50 [unlocked]> is bound to a different event loop

The error seems to always happen (at least the error about using the wrong loop) inside the clean-up code from select(), it might help adding some logging there, like printing a stack trace when the select() was created and when it is being cleaned-up to see if both actions are done in different tests (and different loops).

[llucax]

There seems to be a --setup-show flag that might help checking if we don't somehow have multiple loops overlapping.

Reading a bit more about pytest-asyncio, it seems there have been a few big changes in how this library is supposed to work between 0.21, 0.23 and 0.24. Probably all our code was written using the API 0.21 or older, so maybe issues might be connected to the upgrade to 0.24, although that was done one month ago so I'm not sure it's that likely. But maybe this issue is just surfacing some misconfiguration or misused of 0.24.

Here are official migration guides, it might be worth checking them and making sure we are using pytest-asyncio properly:

• https://pytest-asyncio.readthedocs.io/en/v0.24.0/how-to-guides/migrate_from_0_21.html
• https://pytest-asyncio.readthedocs.io/en/v0.24.0/how-to-guides/migrate_from_0_23.html