Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New asyncio.exceptions.CancelledError from 4.5.2 #2633

Closed
rra opened this issue Mar 21, 2023 · 31 comments
Closed

New asyncio.exceptions.CancelledError from 4.5.2 #2633

rra opened this issue Mar 21, 2023 · 31 comments

Comments

@rra
Copy link

rra commented Mar 21, 2023

Version: 4.5.2
Platform: Python 3.11.2 on Debian unstable
Description:

Starting with 4.5.2, I'm seeing a new exception when attempting to write to Redis using the asyncio layer. The problem seems to be related to FastAPI: if I create a FastAPI app with any Starlette custom middlware, and then try to set a key in Redis using a separate connection pool created with redis.asyncio.from_url, the set request fails immediately with an uncaught asyncio.exceptions.CancelledError and the following backtrace:

.tox/py/lib/python3.11/site-packages/gafaelfawr/storage/base.py:140: in store
    await self._redis.set(key, encrypted_data, ex=lifetime)
.tox/py/lib/python3.11/site-packages/redis/asyncio/client.py:509: in execute_command
    conn = self.connection or await pool.get_connection(command_name, **options)
.tox/py/lib/python3.11/site-packages/redis/asyncio/connection.py:1408: in get_connection
    if await connection.can_read_destructive():
.tox/py/lib/python3.11/site-packages/redis/asyncio/connection.py:817: in can_read_destructive
    return await self._parser.can_read_destructive()
.tox/py/lib/python3.11/site-packages/redis/asyncio/connection.py:250: in can_read_destructive
    return await self._stream.read(1)
/usr/lib/python3.11/asyncio/streams.py:689: in read
    await self._wait_for_data('read')
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <StreamReader transport=<_SelectorSocketTransport fd=15 read=polling write=<idle, bufsize=0>>>
func_name = 'read'

    async def _wait_for_data(self, func_name):
        """Wait until feed_data() or feed_eof() is called.
    
        If stream was paused, automatically resume it.
        """
        # StreamReader uses a future to link the protocol feed_data() method
        # to a read coroutine. Running two read coroutines at the same time
        # would have an unexpected behaviour. It would not possible to know
        # which coroutine would get the next data.
        if self._waiter is not None:
            raise RuntimeError(
                f'{func_name}() called while another coroutine is '
                f'already waiting for incoming data')
    
        assert not self._eof, '_wait_for_data after EOF'
    
        # Waiting for data while paused will make deadlock, so prevent it.
        # This is essential for readexactly(n) for case when n > self._limit.
        if self._paused:
            self._paused = False
            self._transport.resume_reading()
    
        self._waiter = self._loop.create_future()
        try:
>           await self._waiter
E           asyncio.exceptions.CancelledError

/usr/lib/python3.11/asyncio/streams.py:522: CancelledError

GitHub Actions failure log: https://github.com/lsst-sqre/gafaelfawr/actions/runs/4473916874/jobs/7861846517

Reverting to redis 4.5.1 makes the problem disappear again, so it appears to be triggered by some change in 4.5.2.

Here's the shortest test case that I've come up with. It depends on FastAPI and Starlette but none of my code base. My guess is that something is happening during addition of the middleware that's breaking something about the asyncio work inside redis, but I'm not sure what that could be. Oddly, adding one of the standard Starlette middleware classes does not trigger this problem, only a custom middleware class derived from BaseHTTPMiddleware.

from collections.abc import Callable, Awaitable

import redis.asyncio
from fastapi import FastAPI, APIRouter, Request, Response
from httpx import AsyncClient
from starlette.middleware.base import BaseHTTPMiddleware


class DummyMiddleware(BaseHTTPMiddleware):
    async def dispatch(
        self,
        request: Request,
        call_next: Callable[[Request], Awaitable[Response]],
    ) -> Response:
        return await call_next(request)


@pytest.mark.asyncio
async def test_redis() -> None:
    app = FastAPI()
    app.add_middleware(DummyMiddleware)

    @app.get("/")
    async def index() -> dict[str, str]:
        return {}

    async with AsyncClient(app=app, base_url="https://example.com/") as client:
        await client.get("/")
        pool = redis.asyncio.from_url("redis://localhost:6379/0")
        await pool.set("key", "value")

(If I had to guess, it would be the switch to asyncio.timeout, but I can't find evidence to support that. I know asyncio.timeout does raise CancelledError and then converts it to TimeoutError, but I couldn't find any place where that machinery shouldn't work correctly. This error happens immediately, so it doesn't seem to be a timeout and therefore I may be barking up the wrong tree.)

@rra
Copy link
Author

rra commented Mar 21, 2023

Ah, Starlette's BaseHTTPMiddleware calls anyio.create_task_group(). I suspect that is related somehow, although I'm still not seeing exactly how. https://github.com/encode/starlette/blob/master/starlette/middleware/base.py#L39

rra added a commit to lsst-sqre/gafaelfawr that referenced this issue Mar 21, 2023
@sileht
Copy link
Contributor

sileht commented Mar 22, 2023

I guess the CancellationError is raised here, where it should be an asyncio.Timeout.
CleanShot 2023-03-22 at 10 31 26

@sileht
Copy link
Contributor

sileht commented Mar 22, 2023

asyncio check that asyncio.current_task().uncancel() == 0 before converting CancelledError into TimeoutError, so somehow this value is unexpected, not 0.

@sileht
Copy link
Contributor

sileht commented Mar 22, 2023

The thing is that this bug was always there, but with the legacy async_timeout lib, the bug was hidden because of this line:
https://github.com/aio-libs/async-timeout/blob/master/async_timeout/__init__.py#L210

All CancelledError was converted into TimeoutError even if the cancellation source was not the timeout.

@sileht
Copy link
Contributor

sileht commented Mar 22, 2023

Looks related to agronholm/anyio#374

@curtiscook
Copy link

FWIW: I'm experiencing the same issue. I've noticed that the connection does work, but seems to break if you await a function that's not Redis. We were able to reproduce this on the command line w/o starlette (even though we are using FastAPI)
given

def get_redis_pool() -> Redis:
    global _redis_pool

    if _redis_pool is None or _redis_pool.connection is None:
        _redis_pool = aioredis.from_url(REDIS_URL, max_connections=10)
    assert _redis_pool is not None
    return _redis_pool


async def get_key(key: str) -> Any:
    redis = get_redis_pool()
    return await redis.get(key)


async def set_key(key: str, value: Any, expire_seconds: Optional[int] = None) -> bool:
    redis = get_redis_pool()
    if expire_seconds is not None:
        exp = timedelta(seconds=expire_seconds)
        await redis.set(key, value, ex=exp)
    else:
        await redis.set(key, value)
    return True

This function will work fine:

async def foo():
	await get_key("key2")
	await set_key("key2", "value")

This will break:

async def foo():
	await get_key("key2")
	await bar()
	await set_key("key2", "value")

Might be related that bar uses HTTPX

@sileht
Copy link
Contributor

sileht commented Mar 22, 2023

Both starlette and httpx rely on anyio

@rra
Copy link
Author

rra commented Mar 22, 2023

Yeah, this is looking to me like an anyio bug caused by use of task groups, presumably related to this admonition in the Python library documentation:

The asyncio components that enable structured concurrency, like asyncio.TaskGroup and asyncio.timeout(), are implemented using cancellation internally and might misbehave if a coroutine swallows asyncio.CancelledError.

I don't completely understand the mechanism yet, but enough keywords are lining up that this looks like it. @curtiscook's example is interesting, though, since I can't find a call to create_task_group in httpx or httpcore. Maybe it's somewhere else in httpx's dependencies, or maybe that bar function uses anyio?

@rra
Copy link
Author

rra commented Mar 22, 2023

BTW, for those who are specifically having this issue with Starlette middleware, it looks like Starlette has rewritten their built-in middleware to avoid the use of BaseHTTPMiddleware at the cost of having to interact at a lower level with the request and response. That avoids creating a task group, and I wonder if that would make the problem go away. I'm considering rewriting our middleware similarly just as an experiment.

@decaz
Copy link

decaz commented Mar 23, 2023

Looks like version 4.5.3 release didn't resolve the issue.

@bellini666
Copy link
Contributor

FWIW: I'm experiencing the same issue. I've noticed that the connection does work, but seems to break if you await a function that's not Redis. We were able to reproduce this on the command line w/o starlette (even though we are using FastAPI) given

def get_redis_pool() -> Redis:
    global _redis_pool

    if _redis_pool is None or _redis_pool.connection is None:
        _redis_pool = aioredis.from_url(REDIS_URL, max_connections=10)
    assert _redis_pool is not None
    return _redis_pool


async def get_key(key: str) -> Any:
    redis = get_redis_pool()
    return await redis.get(key)


async def set_key(key: str, value: Any, expire_seconds: Optional[int] = None) -> bool:
    redis = get_redis_pool()
    if expire_seconds is not None:
        exp = timedelta(seconds=expire_seconds)
        await redis.set(key, value, ex=exp)
    else:
        await redis.set(key, value)
    return True

This function will work fine:

async def foo():
	await get_key("key2")
	await set_key("key2", "value")

This will break:

async def foo():
	await get_key("key2")
	await bar()
	await set_key("key2", "value")

Might be related that bar uses HTTPX

The same here! I have an async function that will sometimes call an external resource using httpx to validate the input. Everything works fine when httpx is not called, but when it is called then I get CancelledError.

This started happening after upgrading to 4.5.2/4.5.3 (not sure which one exactly since we upgraded from 4.5.1 directly)

@ice1x
Copy link

ice1x commented Mar 23, 2023

Seems like blocker for Redis isn't it?

@sileht
Copy link
Contributor

sileht commented Mar 23, 2023

This bug was always there:

@bellini666
Copy link
Contributor

This bug was always there:

But why does CancelledError gets raised when doing some other async stuff in the same task? E.g. the httpx issue mentioned by me here and by @curtiscook here

I don't understand why CancelledError is being raised when calling a redis api just because I called httx before it (if I don't call httpx, CancelledError is not raised, but when calling, CancelledError` will be raised every time)

I worked around this error by wrapping my call to redis in an asyncio.shield, and although it should be harmless, it really looks like something wrong is happening here.

I'll try to create a minimum reproducible demo

@bellini666
Copy link
Contributor

bellini666 commented Mar 23, 2023

@sileht I have created a minimal reproduction example using httpx and redis@4.5.3

import asyncio

import httpx
import redis.asyncio as redis

pool = redis.ConnectionPool(host="localhost", port=6379, db=0)


async def without_httpx():
    print("without_httpx")
    conn = redis.Redis(connection_pool=pool)
    print(await conn.get("foo"))
    print(await conn.get("bar"))


async def with_httpx_async():
    print("without_httpx_async")
    conn = redis.Redis(connection_pool=pool)
    print(await conn.get("foo"))
    async with httpx.AsyncClient() as client:
        response = await client.get("https://httpbin.org/get")
        print(response.json())
    print(await conn.get("bar"))


async def main():
    conn = redis.Redis(connection_pool=pool)
    await conn.set("foo", "1")
    await conn.set("bar", "2")
    await without_httpx()
    await with_httpx_async()


if __name__ == "__main__":
    asyncio.run(main())

Running it will result in:

without_httpx
b'1'
b'2'
with_httpx_sync
b'1'
{'args': {}, 'headers': {'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'Host': 'httpbin.org', 'User-Agent': 'python-httpx/0.23.3', 'X-Amzn-Trace-Id': 'Root=1-641c81d6-2e145a8c4ca1ee0a178c2573'}, 'origin': '45.4.34.79', 'url': 'https://httpbin.org/get'}
b'2'
without_httpx_async
b'1'
{'args': {}, 'headers': {'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'Host': 'httpbin.org', 'User-Agent': 'python-httpx/0.23.3', 'X-Amzn-Trace-Id': 'Root=1-641c81d6-3ec88c7b5bd6013d2201d553'}, 'origin': '45.4.34.79', 'url': 'https://httpbin.org/get'}
Traceback (most recent call last):
  File "/tmp/xxx/test.py", line 46, in <module>
    asyncio.run(main())
  File "/home/bellini/.local/share/rtx/installs/python/3.11.2/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/home/bellini/.local/share/rtx/installs/python/3.11.2/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bellini/.local/share/rtx/installs/python/3.11.2/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/tmp/xxx/test.py", line 42, in main
    await with_httpx_async()
  File "/tmp/xxx/test.py", line 33, in with_httpx_async
    print(await conn.get("bar"))
          ^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/xxx/venv/lib/python3.11/site-packages/redis/asyncio/client.py", line 509, in execute_command
    conn = self.connection or await pool.get_connection(command_name, **options)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/xxx/venv/lib/python3.11/site-packages/redis/asyncio/connection.py", line 1408, in get_connection
    if await connection.can_read_destructive():
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/xxx/venv/lib/python3.11/site-packages/redis/asyncio/connection.py", line 817, in can_read_destructive
    return await self._parser.can_read_destructive()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/xxx/venv/lib/python3.11/site-packages/redis/asyncio/connection.py", line 250, in can_read_destructive
    return await self._stream.read(1)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bellini/.local/share/rtx/installs/python/3.11.2/lib/python3.11/asyncio/streams.py", line 689, in read
    await self._wait_for_data('read')
  File "/home/bellini/.local/share/rtx/installs/python/3.11.2/lib/python3.11/asyncio/streams.py", line 522, in _wait_for_data
    await self._waiter
asyncio.exceptions.CancelledError

Testing with 4.5.2 also gives the same error, but installing 4.5.1 makes the code work without any errors

In 4.5.1 the output is the expected:

without_httpx
b'1'
b'2'
without_httpx_async
b'1'
{'args': {}, 'headers': {'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'Host': 'httpbin.org', 'User-Agent': 'python-httpx/0.23.3', 'X-Amzn-Trace-Id': 'Root=1-641c825f-044e318c246993f739b4a1b6'}, 'origin': '45.4.34.79', 'url': 'https://httpbin.org/get'}
b'2'

Don't know if httpx is doing something that it shouldn't, but it is strange that it affected redis later without any obvious reason

@curtiscook
Copy link

  • just closed, then reopen.

Since redis uses a connection pool by default, this implies to me that there are two bugs:

  • asyncio.exceptions.CancelledError closing the redis connection (existing)
  • Redis not reconnecting on disconnect when using a connection pool (new)

bellini666 added a commit to bellini666/redis-py that referenced this issue Mar 23, 2023
There's an issue in asyncio's timeout lib before 3.11.3 that causes
async calls to raise `CancelledError`.

This is a cpython issue that was fixed in this commit [1] and
cherry-picked to previous versions, meaning 3.11.3 will work correctly.

Check [2] for more info.

[1] python/cpython@04adf2d
[2] redis#2633
@bellini666
Copy link
Contributor

bellini666 commented Mar 23, 2023

To everyone here, I think I found the culprit.

After testing checking that the issue from my reproduction example did not happen with the async_timeout lib, I went looking at the difference between it and the cpython's one. I didn't have to go too far, when looking at the history from the timeouts module I found this commit: python/cpython@04adf2d

I applied that patch locally and the issue from my reproduction example was gone.

So, this is actually a cpython issue and not a redis issue, and specifically for python 3.11 because for 3.10 and earlier the async_timeout lib is still being used.

What redis can do here is to change all imports:

if sys.version_info.major >= 3 and sys.version_info.minor >= 11:
    from asyncio import timeout as async_timeout
else:
    from async_timeout import timeout as async_timeout

to

if sys.version_info >= (3, 11, 3):
    from asyncio import timeout as async_timeout
else:
    from async_timeout import timeout as async_timeout

Because the change was cherry-picked to the 3.11 branch and will be available on the next point release (which is 3.11.3, the current one is 3.11.2).

I actually already opened a PR for that: #2659

edit: the cpython's PR: python/cpython#102815

@sileht
Copy link
Contributor

sileht commented Mar 23, 2023

@bellini666 nice finding!

@yinziyan1206
Copy link

I have the same problem. :(

dvora-h pushed a commit that referenced this issue Mar 27, 2023
There's an issue in asyncio's timeout lib before 3.11.3 that causes
async calls to raise `CancelledError`.

This is a cpython issue that was fixed in this commit [1] and
cherry-picked to previous versions, meaning 3.11.3 will work correctly.

Check [2] for more info.

[1] python/cpython@04adf2d
[2] #2633
linabutler added a commit to mozilla-services/merino-py that referenced this issue Mar 30, 2023
Redis 4.5.4 has a workaround for redis/redis-py#2633, which was causing
cache misses to raise an uncaught `CancelledError` in the weather
backend.
linabutler added a commit to mozilla-services/merino-py that referenced this issue Mar 30, 2023
Redis 4.5.4 has a workaround for redis/redis-py#2633, which was causing
cache misses to raise an uncaught `CancelledError` in the weather
backend.
@curtiscook
Copy link

Can this be considered fixed as of 4.5.4?

@bellini666
Copy link
Contributor

Can this be considered fixed as of 4.5.4?

I think so. This error was specific to python versions 3.11.0, 3.11.1 and 3.11.2. 4.5.4 only uses python's timeout for python 3.11.3+, which also already got released and doesn't have the issue.

@dvora-h
Copy link
Collaborator

dvora-h commented Apr 30, 2023

@curtiscook Yes, this issue fixed in in 4.5.4 (#2659)

@dvora-h dvora-h closed this as completed Apr 30, 2023
@pinheiro-bruno
Copy link

I'm facing the exactly same issue on version 4.5.5 using python 3.11.2

The issue started after anyio version was bumped from 3.6.2 to 3.7.0. The suggested solution of updating python version to 3.11.3 did the trick for me.

@Mark90
Copy link

Mark90 commented Jun 9, 2023

For me the issue also started to occur after anyio was bumped to 3.7.0.

Posting here because I had a slightly different version configuration;

anyio==3.7.0
redis==4.4.4
python==3.11.4

Upgrading to redis==4.5.5 resolved it for me.

Mark90 added a commit to workfloworchestrator/orchestrator-core that referenced this issue Jun 16, 2023
Mark90 added a commit to workfloworchestrator/orchestrator-core that referenced this issue Jun 20, 2023
pboers1988 pushed a commit to workfloworchestrator/orchestrator-core that referenced this issue Jun 20, 2023
@HacKanCuBa
Copy link

HacKanCuBa commented Jun 22, 2023

With Python 3.11.2, this issue still persists, even using redis==4.5.5 (with hiredis==2.2.3)

Edit: I confirm it all works when upgrading to Python 3.11.4 👯

dvora-h added a commit that referenced this issue Jul 3, 2023
* fix: do not use asyncio's timeout lib before 3.11.2 (#2659)

There's an issue in asyncio's timeout lib before 3.11.3 that causes
async calls to raise `CancelledError`.

This is a cpython issue that was fixed in this commit [1] and
cherry-picked to previous versions, meaning 3.11.3 will work correctly.

Check [2] for more info.

[1] python/cpython@04adf2d
[2] #2633

* UnixDomainSocketConnection missing constructor argument (#2630)

* removing useless files (#2642)

* Fix issue 2660: PytestUnraisableExceptionWarning from asycio client (#2669)

* Fixing cancelled async futures (#2666)

Co-authored-by: James R T <jamestiotio@gmail.com>
Co-authored-by: dvora-h <dvora.heller@redis.com>

* Fix async (#2673)

* Version 4.5.4 (#2674)

* Really do not use asyncio's timeout lib before 3.11.2 (#2699)

4802530 made async-timeout required
only on Python 3.11.2 and earlier. However, according to PEP-508,
python_version marker is compared to first two numbers of Python version
tuple - so it will evaluate to True also on 3.11.3, and install a
package as a dependency.

* asyncio: Fix memory leak caused by hiredis (#2693) (#2694)

* Update example of Redisearch creating index (#2703)

When creating index, fields should be passed inside an iterable (e.g. list or tuple)

* Improving Vector Similarity Search Example (#2661)

* update vss docs

* add embeddings creation and storage examples

* update based on feedback

* fix version and link

* include more realistic search examples and clean up indices

* completely remove initial cap reference

---------

Co-authored-by: Chayim <chayim@users.noreply.github.com>

* Fix incorrect usage of once flag in async Sentinel (#2718)

In the execute_command of the async Sentinel, the once flag was being
used incorrectly, with its meaning inverted. To fix we just needed to invert
the if and else bodies. This isn't being caught by the tests currently
because the tests of commands that use this flag do not check their
results/effects (for example the "test_ckquorum" test).

* Fix topk list example. (#2724)

* Improve error output for master discovery (#2720)

Make MasterNotFoundError exception more precise in the case of
ConnectionError and TimeoutError to help the user to identify
configuration errors

Co-authored-by: Marc Schöchlin <marc.schoechlin@flipapp.de>

* return response in case of KeyError (#2628)

* return response in case of KeyError

* fix code linters error

* fix linters 2

* fix linters 3

* Add WITHSCORES to ZREVRANK Command (#2725)

* add withscores to zrevrank

* change 0 -> 2

* fix errors

* split test

* Fix `ClusterCommandProtocol` not itself being marked as a protocol (#2729)

* Fix `ClusterCommandProtocol` not itself being marked as a protocol

* Update CHANGES

* Fix potential race condition during disconnection (#2719)

When the disconnect() function is called twice in parallel it is possible that
one thread deletes the self._sock reference, while the other thread will
attempt to call .close() on it, leading to an AttributeError.

This situation can routinely be encountered by closing the connection in a
PubSubWorkerThread error handler in a blocking thread (ie. with
sleep_time==None), and then calling .close() on the PubSub object.
The main thread will then run into the disconnect() function, and the listener
thread is woken up by the closure and will race into the disconnect()
function, too.

This can be fixed easily by copying the object reference before doing the
None-check, similar to what we do in the redis.client.close() function.

* add "address_remap" feature to RedisCluster (#2726)

* add cluster "host_port_remap" feature for asyncio.RedisCluster

* Add a unittest for asyncio.RedisCluster

* Add host_port_remap to _sync_ RedisCluster

* add synchronous tests

* rename arg to `address_remap` and take and return an address tuple.

* Add class documentation

* Add CHANGES

* nermina changes from NRedisStack (#2736)

* Updated AWS Elasticache IAM Connection Example (#2702)

Co-authored-by: Nick Gerow <nick.gerow@enlightedinc.com>

* pinning urllib3 to fix CI (#2748)

* Add RedisCluster.remap_host_port, Update tests for CWE 404 (#2706)

* Use provided redis address. Bind to IPv4

* Add missing "await" and perform the correct test for pipe eimpty

* Wait for a send event, rather than rely on sleep time. Excpect cancel errors.

* set delay to 0 except for operation we want to cancel
This speeds up the unit tests considerably by eliminating unnecessary delay.

* Release resources in test

* Fix cluster test to use address_remap and multiple proxies.

* Use context manager to manage DelayProxy

* Mark failing pipeline tests

* lint

* Use a common "master_host" test fixture

* Update redismodules.rst (#2747)

Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>

* Add support for cluster myshardid (#2704)

* feat: adding support for cluster myshardid

* lint fix

* fix: comment fix and async test

* fix: adding version check

* fix lint:

* linters

---------

Co-authored-by: Anuragkillswitch <70265851+Anuragkillswitch@users.noreply.github.com>
Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>
Co-authored-by: dvora-h <dvora.heller@redis.com>

* clean warnings (#2731)

* fix parse_slowlog_get (#2732)

* Optionally disable disconnects in read_response (#2695)

* Add regression tests and fixes for issue #1128

* Fix tests for resumable read_response to use "disconnect_on_error"

* undo prevision fix attempts in async client and cluster

* re-enable cluster test

* Suggestions from code review

* Add CHANGES

* Add client no-touch (#2745)

* Add client no-touch

* Update redis/commands/core.py

Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>

* Update test_commands.py

Improve test_client_no_touch

* Update test_commands.py

Add async version test case

* Chore remove whitespace

Oops

---------

Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>

* fix create single_connection_client from url (#2752)

* Fix `xadd` allow non negative maxlen (#2739)

* Fix xadd allow non negative maxlen

* Update change log

---------

Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>

* Version 4.5.5 (#2753)

* Kristjan/issue #2754: Add missing argument to SentinelManagedConnection.read_response() (#2756)

* Increase timeout for a test which would hang completely if failing.
Timeouts in virtualized CI backends can occasionally fail if too short.

* add "disconnect_on_error" argument to SentinelManagedConnection

* update Changes

* lint

* support JSON.MERGE Command (#2761)

* support JSON.MERGE Command

* linters

* try with abc instead person

* change @skip_ifmodversion_lt to latest ReJSON 2.4.7

* change version

* fix test

* linters

* add async test

* Issue #2749: Remove unnecessary __del__ handlers (#2755)

* Remove unnecessary __del__ handlers
There normally should be no logic attached to del.  Cleanly disconnecting network resources is not needed at that time.

* add CHANGES

* Add WITHSCORE to ZRANK (#2758)

* add withscore to zrank with tests

* fix test

* Fix JSON.MERGE Summary (#2786)

* Fix JSON.MERGE Summary

* linters

* Fixed key error in parse_xinfo_stream (#2788)

* insert newline to prevent sphinx from assuming code block (#2796)

* Introduce OutOfMemoryError exception for Redis write command rejections due to OOM errors (#2778)

* expose OutOfMemoryError as explicit exception type

- handle "OOM" error code string by raising explicit
  exception type instance
- enables callers to avoid string matching after
  catching ResponseError

* add OutOfMemoryError exception class docstring

* Provide more info in the exception docstring

* Fix formatting

* Again

* linters

---------

Co-authored-by: Chayim <chayim@users.noreply.github.com>
Co-authored-by: Igor Malinovskiy <u.glide@gmail.com>
Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>

* Add unit tests for the `connect` method of all Redis connection classes (#2631)

* tests: move certificate discovery to a separate module

* tests: add 'connect' tests for all Redis connection classes

---------

Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>

* Fix dead weakref in sentinel connection causing ReferenceError (#2767) (#2771)

* Fix dead weakref in sentinel conn (#2767)

* Update CHANGES

---------

Co-authored-by: Igor Malinovskiy <u.glide@gmail.com>
Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>

* chore(documentation): fix redirects and some small cleanups (#2801)

* Add waitaof (#2760)

* Add waitaof

* Update test_commands.py

add test_waitaof

* Update test_commands.py

Add test_waitaof

* Fix doc string

---------

Co-authored-by: Chayim <chayim@users.noreply.github.com>
Co-authored-by: Igor Malinovskiy <u.glide@gmail.com>

* Extract abstract async connection class (#2734)

* make 'socket_timeout' and 'socket_connect_timeout' equivalent for TCP and UDS connections

* abstract asynio connection in analogy with the synchronous connection

---------

Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>

* Fix type hint for retry_on_error in async cluster (#2804)

* fix(asyncio.cluster): fixup retry_on_error type hint

This parameter accepts a list of _classes of Exceptions_, not a list of instantiated Exceptions. Fixup the type hint accordingly.

* chore: update changelog

---------

Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>

* Fix CI (#2809)

* Support JSON.MSET Command (#2766)

* support JSON.MERGE Command

* linters

* try with abc instead person

* change @skip_ifmodversion_lt to latest ReJSON 2.4.7

* change version

* fix test

* linters

* add async test

* Support JSON.MSET command

* trying to run CI

* linters

* add async test

* reminder do delete the integration changes

* delete the line from integration

* fix the interface

* change docstring

---------

Co-authored-by: Chayim <chayim@users.noreply.github.com>
Co-authored-by: dvora-h <dvora.heller@redis.com>

* Version 4.6.0 (#2810)

* master changes

* linters

* fix test_cwe_404 cluster test

---------

Co-authored-by: Thiago Bellini Ribeiro <hackedbellini@gmail.com>
Co-authored-by: woutdenolf <woutdenolf@users.sf.net>
Co-authored-by: Chayim <chayim@users.noreply.github.com>
Co-authored-by: shacharPash <93581407+shacharPash@users.noreply.github.com>
Co-authored-by: James R T <jamestiotio@gmail.com>
Co-authored-by: Mirek Długosz <miniopl+github@gmail.com>
Co-authored-by: Oran Avraham <252748+oranav@users.noreply.github.com>
Co-authored-by: mzdehbashi-github <85902780+mzdehbashi-github@users.noreply.github.com>
Co-authored-by: Tyler Hutcherson <tyler.hutcherson@redis.com>
Co-authored-by: Felipe Machado <462154+felipou@users.noreply.github.com>
Co-authored-by: AYMEN Mohammed <53928879+AYMENJD@users.noreply.github.com>
Co-authored-by: Marc Schöchlin <ms-github@256bit.org>
Co-authored-by: Marc Schöchlin <marc.schoechlin@flipapp.de>
Co-authored-by: Avasam <samuel.06@hotmail.com>
Co-authored-by: Markus Gerstel <2102431+Anthchirp@users.noreply.github.com>
Co-authored-by: Kristján Valur Jónsson <sweskman@gmail.com>
Co-authored-by: Nick Gerow <Nick.G.123@hotmail.com>
Co-authored-by: Nick Gerow <nick.gerow@enlightedinc.com>
Co-authored-by: Cristian Matache <cristianmatache@hotmail.com>
Co-authored-by: Anurag Bandyopadhyay <angbpy@gmail.com>
Co-authored-by: Anuragkillswitch <70265851+Anuragkillswitch@users.noreply.github.com>
Co-authored-by: Seongchuel Ahn <aciddust20@gmail.com>
Co-authored-by: Alibi <aliby.bbb@gmail.com>
Co-authored-by: Smit Parmar <smitraj333@gmail.com>
Co-authored-by: Brad MacPhee <macphee@gmail.com>
Co-authored-by: Igor Malinovskiy <u.glide@gmail.com>
Co-authored-by: Shahar Lev <shahar_lev@hotmail.com>
Co-authored-by: Vladimir Mihailenco <vladimir.webdev@gmail.com>
Co-authored-by: Kevin James <KevinJames@thekev.in>
@bsnacks000
Copy link

We had the same problem... resolved with 3.11.4 bump from 3.11.2... thanks for figuring that out ... 👍

python = "^3.11.4"
redis = "^4.6.0"
httpx = "0.23.0"
anyio = "^3.7.1"

dvora-h added a commit that referenced this issue Jul 16, 2023
* Reorganizing the parsers code, and add support for RESP3 (#2574)

* Reorganizing the parsers code

* fix build package

* fix imports

* fix flake8

* add resp to Connection class

* core commands

* python resp3 parser

* pipeline

* async resp3 parser

* some asymc tests

* resp3 parser for async cluster

* async commands tests

* linters

* linters

* linters

* fix ModuleNotFoundError

* fix tests

* fix assert_resp_response_in

* fix command_getkeys in cluster

* fail-fast false

* version

---------

Co-authored-by: Chayim I. Kirshen <c@kirshen.com>

* Fix async client with resp3 (#2657)

* Add support for PubSub with RESP3 parser (#2721)

* add resp3 pubsub

* linters

* _set_info_logger func

* async pubsun

* docstring

* 5.0.0b2 (#2723)

* Fix `COMMAND` response in resp3 (redis 7+) (#2740)

* Fix protocol version checking (#2737)

* bumping beta version to 5.0.0b3 (#2743)

* Fix parse resp3 dict response: don't use dict comprehension (#2757)

* Fix parse respp3 dict response

* linters

* pin urlib version

* Sharded pubsub (#2762)

* sharded pubsub

* sharded pubsub

Co-authored-by: Leibale Eidelman <me@leibale.com>

* Shrded Pubsub TestPubSubSubscribeUnsubscribe

* fix TestPubSubSubscribeUnsubscribe

* more tests

* linters

* TestPubSubSubcommands

* fix @leibale comments

* linters

* fix @chayim comments

---------

Co-authored-by: Leibale Eidelman <me@leibale.com>

* 5.0.0b4 (#2781)

Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>

* RESP3 tests (#2780)

* fix command response in resp3

* linters

* acl_log & acl_getuser

* client_info

* test_commands and test_asyncio/test_commands

* fix test_command_parser

* fix asyncio/test_connection/test_invalid_response

* linters

* all the tests

* push handler sharded pubsub

* Use assert_resp_response wherever possible

* fix test_xreadgroup

* fix cluster_zdiffstore and cluster_zinter

* fix review comments

* fix review comments

* linters

* Fixing asyncio import (#2759)

* asyncio import fix

* pinning urllib3 to fix CI (#2748)

* noqa

* fixint linters

* fix (#2799)

* RESP3 response callbacks (#2798)

* start cleaning

* clean sone callbacks

* response callbacks

* revert redismod-url change

* fix async tests

* linters

* async cluster

---------

Co-authored-by: Chayim <chayim@users.noreply.github.com>

* RESP3 modules support (#2803)

* start cleaning

* clean sone callbacks

* response callbacks

* modules

* tests

* finish sync search tests

* linters

* async modules

* linters

* revert redismod-url change

* RESP3 fix async tests (#2806)

* fix tests

* add stralgo callback in resp2

* add callback to acl list in resp2

* Adding RESP3 tests support (#2793)

* start cleaning

* clean sone callbacks

* first phase

* tox wrap back

* changing cancel format

* syntax

* lint

* docker

* contain the docker

* tox dev reqs

* back to testing

* response callbacks

* protocol into async conftest

* fix for 3.11 invoke

* docker changes

* fix tests

* linters

* adding

* resp3 tox, until killed

* remove tox

* tests

* requirements.txt

* restoring requirements.txt

* adding a sleep, hopefully enough time for the cluster dockers to settle

* fix search tests

* search test, disable uvloop for pypy due to bug

* syn

* reg

* dialect test improvement

* sleep+, xfail

* tests

* resp

* flaky search test too

* timing

* timing for async test

* test changes

* fix assert_interval_advanced

* revert

* mark async health_check tests with xfail

* change strict to false

* fix github actions package validation

---------

Co-authored-by: dvora-h <dvora.heller@redis.com>

* change sismember return type (#2813)

* Version 5.0.0rc1 (#2815)

* Merge master to 5.0 (#2827)

* fix: do not use asyncio's timeout lib before 3.11.2 (#2659)

There's an issue in asyncio's timeout lib before 3.11.3 that causes
async calls to raise `CancelledError`.

This is a cpython issue that was fixed in this commit [1] and
cherry-picked to previous versions, meaning 3.11.3 will work correctly.

Check [2] for more info.

[1] python/cpython@04adf2d
[2] #2633

* UnixDomainSocketConnection missing constructor argument (#2630)

* removing useless files (#2642)

* Fix issue 2660: PytestUnraisableExceptionWarning from asycio client (#2669)

* Fixing cancelled async futures (#2666)

Co-authored-by: James R T <jamestiotio@gmail.com>
Co-authored-by: dvora-h <dvora.heller@redis.com>

* Fix async (#2673)

* Version 4.5.4 (#2674)

* Really do not use asyncio's timeout lib before 3.11.2 (#2699)

4802530 made async-timeout required
only on Python 3.11.2 and earlier. However, according to PEP-508,
python_version marker is compared to first two numbers of Python version
tuple - so it will evaluate to True also on 3.11.3, and install a
package as a dependency.

* asyncio: Fix memory leak caused by hiredis (#2693) (#2694)

* Update example of Redisearch creating index (#2703)

When creating index, fields should be passed inside an iterable (e.g. list or tuple)

* Improving Vector Similarity Search Example (#2661)

* update vss docs

* add embeddings creation and storage examples

* update based on feedback

* fix version and link

* include more realistic search examples and clean up indices

* completely remove initial cap reference

---------

Co-authored-by: Chayim <chayim@users.noreply.github.com>

* Fix incorrect usage of once flag in async Sentinel (#2718)

In the execute_command of the async Sentinel, the once flag was being
used incorrectly, with its meaning inverted. To fix we just needed to invert
the if and else bodies. This isn't being caught by the tests currently
because the tests of commands that use this flag do not check their
results/effects (for example the "test_ckquorum" test).

* Fix topk list example. (#2724)

* Improve error output for master discovery (#2720)

Make MasterNotFoundError exception more precise in the case of
ConnectionError and TimeoutError to help the user to identify
configuration errors

Co-authored-by: Marc Schöchlin <marc.schoechlin@flipapp.de>

* return response in case of KeyError (#2628)

* return response in case of KeyError

* fix code linters error

* fix linters 2

* fix linters 3

* Add WITHSCORES to ZREVRANK Command (#2725)

* add withscores to zrevrank

* change 0 -> 2

* fix errors

* split test

* Fix `ClusterCommandProtocol` not itself being marked as a protocol (#2729)

* Fix `ClusterCommandProtocol` not itself being marked as a protocol

* Update CHANGES

* Fix potential race condition during disconnection (#2719)

When the disconnect() function is called twice in parallel it is possible that
one thread deletes the self._sock reference, while the other thread will
attempt to call .close() on it, leading to an AttributeError.

This situation can routinely be encountered by closing the connection in a
PubSubWorkerThread error handler in a blocking thread (ie. with
sleep_time==None), and then calling .close() on the PubSub object.
The main thread will then run into the disconnect() function, and the listener
thread is woken up by the closure and will race into the disconnect()
function, too.

This can be fixed easily by copying the object reference before doing the
None-check, similar to what we do in the redis.client.close() function.

* add "address_remap" feature to RedisCluster (#2726)

* add cluster "host_port_remap" feature for asyncio.RedisCluster

* Add a unittest for asyncio.RedisCluster

* Add host_port_remap to _sync_ RedisCluster

* add synchronous tests

* rename arg to `address_remap` and take and return an address tuple.

* Add class documentation

* Add CHANGES

* nermina changes from NRedisStack (#2736)

* Updated AWS Elasticache IAM Connection Example (#2702)

Co-authored-by: Nick Gerow <nick.gerow@enlightedinc.com>

* pinning urllib3 to fix CI (#2748)

* Add RedisCluster.remap_host_port, Update tests for CWE 404 (#2706)

* Use provided redis address. Bind to IPv4

* Add missing "await" and perform the correct test for pipe eimpty

* Wait for a send event, rather than rely on sleep time. Excpect cancel errors.

* set delay to 0 except for operation we want to cancel
This speeds up the unit tests considerably by eliminating unnecessary delay.

* Release resources in test

* Fix cluster test to use address_remap and multiple proxies.

* Use context manager to manage DelayProxy

* Mark failing pipeline tests

* lint

* Use a common "master_host" test fixture

* Update redismodules.rst (#2747)

Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>

* Add support for cluster myshardid (#2704)

* feat: adding support for cluster myshardid

* lint fix

* fix: comment fix and async test

* fix: adding version check

* fix lint:

* linters

---------

Co-authored-by: Anuragkillswitch <70265851+Anuragkillswitch@users.noreply.github.com>
Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>
Co-authored-by: dvora-h <dvora.heller@redis.com>

* clean warnings (#2731)

* fix parse_slowlog_get (#2732)

* Optionally disable disconnects in read_response (#2695)

* Add regression tests and fixes for issue #1128

* Fix tests for resumable read_response to use "disconnect_on_error"

* undo prevision fix attempts in async client and cluster

* re-enable cluster test

* Suggestions from code review

* Add CHANGES

* Add client no-touch (#2745)

* Add client no-touch

* Update redis/commands/core.py

Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>

* Update test_commands.py

Improve test_client_no_touch

* Update test_commands.py

Add async version test case

* Chore remove whitespace

Oops

---------

Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>

* fix create single_connection_client from url (#2752)

* Fix `xadd` allow non negative maxlen (#2739)

* Fix xadd allow non negative maxlen

* Update change log

---------

Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>

* Version 4.5.5 (#2753)

* Kristjan/issue #2754: Add missing argument to SentinelManagedConnection.read_response() (#2756)

* Increase timeout for a test which would hang completely if failing.
Timeouts in virtualized CI backends can occasionally fail if too short.

* add "disconnect_on_error" argument to SentinelManagedConnection

* update Changes

* lint

* support JSON.MERGE Command (#2761)

* support JSON.MERGE Command

* linters

* try with abc instead person

* change @skip_ifmodversion_lt to latest ReJSON 2.4.7

* change version

* fix test

* linters

* add async test

* Issue #2749: Remove unnecessary __del__ handlers (#2755)

* Remove unnecessary __del__ handlers
There normally should be no logic attached to del.  Cleanly disconnecting network resources is not needed at that time.

* add CHANGES

* Add WITHSCORE to ZRANK (#2758)

* add withscore to zrank with tests

* fix test

* Fix JSON.MERGE Summary (#2786)

* Fix JSON.MERGE Summary

* linters

* Fixed key error in parse_xinfo_stream (#2788)

* insert newline to prevent sphinx from assuming code block (#2796)

* Introduce OutOfMemoryError exception for Redis write command rejections due to OOM errors (#2778)

* expose OutOfMemoryError as explicit exception type

- handle "OOM" error code string by raising explicit
  exception type instance
- enables callers to avoid string matching after
  catching ResponseError

* add OutOfMemoryError exception class docstring

* Provide more info in the exception docstring

* Fix formatting

* Again

* linters

---------

Co-authored-by: Chayim <chayim@users.noreply.github.com>
Co-authored-by: Igor Malinovskiy <u.glide@gmail.com>
Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>

* Add unit tests for the `connect` method of all Redis connection classes (#2631)

* tests: move certificate discovery to a separate module

* tests: add 'connect' tests for all Redis connection classes

---------

Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>

* Fix dead weakref in sentinel connection causing ReferenceError (#2767) (#2771)

* Fix dead weakref in sentinel conn (#2767)

* Update CHANGES

---------

Co-authored-by: Igor Malinovskiy <u.glide@gmail.com>
Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>

* chore(documentation): fix redirects and some small cleanups (#2801)

* Add waitaof (#2760)

* Add waitaof

* Update test_commands.py

add test_waitaof

* Update test_commands.py

Add test_waitaof

* Fix doc string

---------

Co-authored-by: Chayim <chayim@users.noreply.github.com>
Co-authored-by: Igor Malinovskiy <u.glide@gmail.com>

* Extract abstract async connection class (#2734)

* make 'socket_timeout' and 'socket_connect_timeout' equivalent for TCP and UDS connections

* abstract asynio connection in analogy with the synchronous connection

---------

Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>

* Fix type hint for retry_on_error in async cluster (#2804)

* fix(asyncio.cluster): fixup retry_on_error type hint

This parameter accepts a list of _classes of Exceptions_, not a list of instantiated Exceptions. Fixup the type hint accordingly.

* chore: update changelog

---------

Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>

* Fix CI (#2809)

* Support JSON.MSET Command (#2766)

* support JSON.MERGE Command

* linters

* try with abc instead person

* change @skip_ifmodversion_lt to latest ReJSON 2.4.7

* change version

* fix test

* linters

* add async test

* Support JSON.MSET command

* trying to run CI

* linters

* add async test

* reminder do delete the integration changes

* delete the line from integration

* fix the interface

* change docstring

---------

Co-authored-by: Chayim <chayim@users.noreply.github.com>
Co-authored-by: dvora-h <dvora.heller@redis.com>

* Version 4.6.0 (#2810)

* master changes

* linters

* fix test_cwe_404 cluster test

---------

Co-authored-by: Thiago Bellini Ribeiro <hackedbellini@gmail.com>
Co-authored-by: woutdenolf <woutdenolf@users.sf.net>
Co-authored-by: Chayim <chayim@users.noreply.github.com>
Co-authored-by: shacharPash <93581407+shacharPash@users.noreply.github.com>
Co-authored-by: James R T <jamestiotio@gmail.com>
Co-authored-by: Mirek Długosz <miniopl+github@gmail.com>
Co-authored-by: Oran Avraham <252748+oranav@users.noreply.github.com>
Co-authored-by: mzdehbashi-github <85902780+mzdehbashi-github@users.noreply.github.com>
Co-authored-by: Tyler Hutcherson <tyler.hutcherson@redis.com>
Co-authored-by: Felipe Machado <462154+felipou@users.noreply.github.com>
Co-authored-by: AYMEN Mohammed <53928879+AYMENJD@users.noreply.github.com>
Co-authored-by: Marc Schöchlin <ms-github@256bit.org>
Co-authored-by: Marc Schöchlin <marc.schoechlin@flipapp.de>
Co-authored-by: Avasam <samuel.06@hotmail.com>
Co-authored-by: Markus Gerstel <2102431+Anthchirp@users.noreply.github.com>
Co-authored-by: Kristján Valur Jónsson <sweskman@gmail.com>
Co-authored-by: Nick Gerow <Nick.G.123@hotmail.com>
Co-authored-by: Nick Gerow <nick.gerow@enlightedinc.com>
Co-authored-by: Cristian Matache <cristianmatache@hotmail.com>
Co-authored-by: Anurag Bandyopadhyay <angbpy@gmail.com>
Co-authored-by: Anuragkillswitch <70265851+Anuragkillswitch@users.noreply.github.com>
Co-authored-by: Seongchuel Ahn <aciddust20@gmail.com>
Co-authored-by: Alibi <aliby.bbb@gmail.com>
Co-authored-by: Smit Parmar <smitraj333@gmail.com>
Co-authored-by: Brad MacPhee <macphee@gmail.com>
Co-authored-by: Igor Malinovskiy <u.glide@gmail.com>
Co-authored-by: Shahar Lev <shahar_lev@hotmail.com>
Co-authored-by: Vladimir Mihailenco <vladimir.webdev@gmail.com>
Co-authored-by: Kevin James <KevinJames@thekev.in>

* RESP3 response-callbacks cleanup (#2841)

* cluenup

* sentinel callbacks

* move callbacks

* fix async cluster tests

* _parsers and import fix in tests

* linters

* make modules callbacks private

* fix async search

* fix

---------

Co-authored-by: Chayim I. Kirshen <c@kirshen.com>

* Version 5.0.0rc2 (#2843)

* linters

---------

Co-authored-by: Chayim I. Kirshen <c@kirshen.com>
Co-authored-by: Chayim <chayim@users.noreply.github.com>
Co-authored-by: Leibale Eidelman <me@leibale.com>
Co-authored-by: Thiago Bellini Ribeiro <hackedbellini@gmail.com>
Co-authored-by: woutdenolf <woutdenolf@users.sf.net>
Co-authored-by: shacharPash <93581407+shacharPash@users.noreply.github.com>
Co-authored-by: James R T <jamestiotio@gmail.com>
Co-authored-by: Mirek Długosz <miniopl+github@gmail.com>
Co-authored-by: Oran Avraham <252748+oranav@users.noreply.github.com>
Co-authored-by: mzdehbashi-github <85902780+mzdehbashi-github@users.noreply.github.com>
Co-authored-by: Tyler Hutcherson <tyler.hutcherson@redis.com>
Co-authored-by: Felipe Machado <462154+felipou@users.noreply.github.com>
Co-authored-by: AYMEN Mohammed <53928879+AYMENJD@users.noreply.github.com>
Co-authored-by: Marc Schöchlin <ms-github@256bit.org>
Co-authored-by: Marc Schöchlin <marc.schoechlin@flipapp.de>
Co-authored-by: Avasam <samuel.06@hotmail.com>
Co-authored-by: Markus Gerstel <2102431+Anthchirp@users.noreply.github.com>
Co-authored-by: Kristján Valur Jónsson <sweskman@gmail.com>
Co-authored-by: Nick Gerow <Nick.G.123@hotmail.com>
Co-authored-by: Nick Gerow <nick.gerow@enlightedinc.com>
Co-authored-by: Cristian Matache <cristianmatache@hotmail.com>
Co-authored-by: Anurag Bandyopadhyay <angbpy@gmail.com>
Co-authored-by: Anuragkillswitch <70265851+Anuragkillswitch@users.noreply.github.com>
Co-authored-by: Seongchuel Ahn <aciddust20@gmail.com>
Co-authored-by: Alibi <aliby.bbb@gmail.com>
Co-authored-by: Smit Parmar <smitraj333@gmail.com>
Co-authored-by: Brad MacPhee <macphee@gmail.com>
Co-authored-by: Igor Malinovskiy <u.glide@gmail.com>
Co-authored-by: Shahar Lev <shahar_lev@hotmail.com>
Co-authored-by: Vladimir Mihailenco <vladimir.webdev@gmail.com>
Co-authored-by: Kevin James <KevinJames@thekev.in>
@hendrixmar
Copy link

Thanks for the help it worked upgrading to 3.11.4

@Naltharial
Copy link

@dvora-h Apologies for resurrecting an old issue, but it seems like it's relevant again. I'm seeing this happen on 3.11.8 with AnyIO 4, PyRedis 5 using @bellini666 's test script.

Using python3.11 (3.11.8)
without_httpx
b'1'
b'2'
with_httpx_async
b'1'
{'args': {}, 'headers': {'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'Host': 'httpbin.org', 'User-Agent': 'python-httpx/0.27.0', 'X-Amzn-Trace-Id': 'Root=1-660b04d4-0ed17ec26d7b388724aaf03b'}, 'origin': '212.85.182.26', 'url': 'https://httpbin.org/get'}
Traceback (most recent call last):
  File "/mnt/d/Development/Infold/assembler/asyn_bug.py", line 35, in <module>
    asyncio.run(main())
  File "/usr/lib/python3.11/asyncio/runners.py", line 188, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/asyncio/runners.py", line 120, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/asyncio/base_events.py", line 650, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/mnt/d/Development/Infold/assembler/asyn_bug.py", line 31, in main
    await with_httpx_async()
  File "/mnt/d/Development/Infold/assembler/asyn_bug.py", line 23, in with_httpx_async
    print(await conn.get("bar"))
          ^^^^^^^^^^^^^^^^^^^^^
  File "/home/naltharial/.cache/pypoetry/virtualenvs/assembler-01ENdme1-py3.11/lib/python3.11/site-packages/redis/asyncio/client.py", line 605, in execute_command
    conn = self.connection or await pool.get_connection(command_name, **options)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/naltharial/.cache/pypoetry/virtualenvs/assembler-01ENdme1-py3.11/lib/python3.11/site-packages/redis/asyncio/connection.py", line 1076, in get_connection
    await self.ensure_connection(connection)
  File "/home/naltharial/.cache/pypoetry/virtualenvs/assembler-01ENdme1-py3.11/lib/python3.11/site-packages/redis/asyncio/connection.py", line 1115, in ensure_connection
    if await connection.can_read_destructive():
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/naltharial/.cache/pypoetry/virtualenvs/assembler-01ENdme1-py3.11/lib/python3.11/site-packages/redis/asyncio/connection.py", line 504, in can_read_destructive
    return await self._parser.can_read_destructive()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/naltharial/.cache/pypoetry/virtualenvs/assembler-01ENdme1-py3.11/lib/python3.11/site-packages/redis/_parsers/base.py", line 185, in can_read_destructive
    return await self._stream.read(1)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/asyncio/streams.py", line 690, in read
    await self._wait_for_data('read')
  File "/usr/lib/python3.11/asyncio/streams.py", line 523, in _wait_for_data
    await self._waiter
asyncio.exceptions.CancelledError
Exception ignored in: <function AbstractConnection.__del__ at 0x7f06a59e40e0>
Traceback (most recent call last):
  File "/home/naltharial/.cache/pypoetry/virtualenvs/assembler-01ENdme1-py3.11/lib/python3.11/site-packages/redis/asyncio/connection.py", line 216, in __del__
  File "/home/naltharial/.cache/pypoetry/virtualenvs/assembler-01ENdme1-py3.11/lib/python3.11/site-packages/redis/asyncio/connection.py", line 223, in _close
  File "/usr/lib/python3.11/asyncio/streams.py", line 345, in close
  File "/usr/lib/python3.11/asyncio/selector_events.py", line 831, in close
  File "/usr/lib/python3.11/asyncio/base_events.py", line 758, in call_soon
  File "/usr/lib/python3.11/asyncio/base_events.py", line 519, in _check_closed
RuntimeError: Event loop is closed

it looks like HTTPx is closing the event loop behind it, no matter if Redis is still trying to use it. Is this related to the same issue? Notably, this only occurs under Ubuntu, I can't reproduce it on Windows, so my debugging capacity is limited.

It's resolved (obviously) by using the sync Redis API instead of the async one, but upgrading the libraries didn't work.

Package                       Version
----------------------------- -----------
...
anyio                         4.3.0
httpx                         0.27.0
redis                         5.0.3
trio                          0.25.0
...

@Naltharial
Copy link

Naltharial commented Apr 2, 2024

The event loop closing might be a red herring, this is the output on Py3.11.8 / Windows:

b'1'
b'2'
with_httpx_async
b'1'
{'args': {}, 'headers': {'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'Host': 'httpbin.org', 'User-Agent': 'python-httpx/0.27.0', 'X-Amzn-Trace-Id': 'Root=1-660bc9f9-43ac10016bb2539865135414'}, 'origin': '212.85.182.26', 'url': 'https://httpbin.org/get'}
b'2'
Exception ignored in: <function AbstractConnection.__del__ at 0x000002D87C3D1620>
Traceback (most recent call last):
  File "C:\Users\Uporabnik\AppData\Local\pypoetry\Cache\virtualenvs\assembler-JBayZ5rl-py3.11\Lib\site-packages\redis\asyncio\connection.py", line 216, in __del__
  File "C:\Users\Uporabnik\AppData\Local\pypoetry\Cache\virtualenvs\assembler-JBayZ5rl-py3.11\Lib\site-packages\redis\asyncio\connection.py", line 223, in _close
  File "C:\Python311\Lib\asyncio\streams.py", line 358, in close
  File "C:\Python311\Lib\asyncio\proactor_events.py", line 109, in close
  File "C:\Python311\Lib\asyncio\base_events.py", line 762, in call_soon
  File "C:\Python311\Lib\asyncio\base_events.py", line 520, in _check_closed
RuntimeError: Event loop is closed

The b'2' comes through and no CancelledError, but something is still trying to access the event loop from Redis even past the end. The usual culprit for Linux/Windows async inconsistencies is uvloop, but its presence didn't seem to change anything.

@Mark90
Copy link

Mark90 commented Apr 2, 2024

I'm not hitting this issue right now, but I got curious and I can also reproduce it with the latest test script from @bellini666. I tested a few more scenarios.
Note that I'm running Windows, but this is tested with Ubuntu 20.04 in Windows Subsystem Linux (basically a VM).

For each python version I tested with, I started with redis-py 5.0.3 and downgraded through 5.0.2 -> 5.0.1 -> 5.0.0 -> 4.6.0 to see if/when behavior changed. The envs were clean and I did not install trio nor uvloop.

python anyio httpx redis result
3.11.1 4.3.0 0.27.0 5.0.0-5.0.3 CE + RE on conn.get() after httpx
3.11.1 4.3.0 0.27.0 4.6.0 works
3.11.8 4.3.0 0.27.0 5.0.2, 5.0.3 works, but RE on shutdown
3.11.8 4.3.0 0.27.0 4.6.0, 5.0.0, 5.0.1 works
3.12.1 4.3.0 0.27.0 5.0.2, 5.0.3 works, but RE on shutdown
3.12.1 4.3.0 0.27.0 4.6.0, 5.0.0, 5.0.1 works

CE = asyncio.exceptions.CancelledError (same as @Naltharial)
RE = RuntimeError: Event loop is closed (Exception ignored in AbstractConnection.del, same as @Naltharial)

Some other observations:

  • In the case of 3.11.1, moving the conn.get() into the with AsyncClient block did not trigger the issue as long as it comes before the client.get() call. Right below the client.get() call (within the context) is where the problem starts
  • I tried swapping httpx out for aiohttp and this does not trigger the issue, so somehow httpx and redis-py bite (could still be a be a timing-related coincidence)

Because on 3.11.8/3.12.1 the behavior seems to start from redis-py 5.0.2 onwards, perhaps #2999 could be related? (edit: except that does not explain the combination 3.11.1 and redis-py 5.0.0..)

@yinziyan1206
Copy link

@dvora-h Apologies for resurrecting an old issue, but it seems like it's relevant again. I'm seeing this happen on 3.11.8 with AnyIO 4, PyRedis 5 using @bellini666 's test script.

Using python3.11 (3.11.8)
without_httpx
b'1'
b'2'
with_httpx_async
b'1'
{'args': {}, 'headers': {'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'Host': 'httpbin.org', 'User-Agent': 'python-httpx/0.27.0', 'X-Amzn-Trace-Id': 'Root=1-660b04d4-0ed17ec26d7b388724aaf03b'}, 'origin': '212.85.182.26', 'url': 'https://httpbin.org/get'}
Traceback (most recent call last):
  File "/mnt/d/Development/Infold/assembler/asyn_bug.py", line 35, in <module>
    asyncio.run(main())
  File "/usr/lib/python3.11/asyncio/runners.py", line 188, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/asyncio/runners.py", line 120, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/asyncio/base_events.py", line 650, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/mnt/d/Development/Infold/assembler/asyn_bug.py", line 31, in main
    await with_httpx_async()
  File "/mnt/d/Development/Infold/assembler/asyn_bug.py", line 23, in with_httpx_async
    print(await conn.get("bar"))
          ^^^^^^^^^^^^^^^^^^^^^
  File "/home/naltharial/.cache/pypoetry/virtualenvs/assembler-01ENdme1-py3.11/lib/python3.11/site-packages/redis/asyncio/client.py", line 605, in execute_command
    conn = self.connection or await pool.get_connection(command_name, **options)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/naltharial/.cache/pypoetry/virtualenvs/assembler-01ENdme1-py3.11/lib/python3.11/site-packages/redis/asyncio/connection.py", line 1076, in get_connection
    await self.ensure_connection(connection)
  File "/home/naltharial/.cache/pypoetry/virtualenvs/assembler-01ENdme1-py3.11/lib/python3.11/site-packages/redis/asyncio/connection.py", line 1115, in ensure_connection
    if await connection.can_read_destructive():
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/naltharial/.cache/pypoetry/virtualenvs/assembler-01ENdme1-py3.11/lib/python3.11/site-packages/redis/asyncio/connection.py", line 504, in can_read_destructive
    return await self._parser.can_read_destructive()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/naltharial/.cache/pypoetry/virtualenvs/assembler-01ENdme1-py3.11/lib/python3.11/site-packages/redis/_parsers/base.py", line 185, in can_read_destructive
    return await self._stream.read(1)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/asyncio/streams.py", line 690, in read
    await self._wait_for_data('read')
  File "/usr/lib/python3.11/asyncio/streams.py", line 523, in _wait_for_data
    await self._waiter
asyncio.exceptions.CancelledError
Exception ignored in: <function AbstractConnection.__del__ at 0x7f06a59e40e0>
Traceback (most recent call last):
  File "/home/naltharial/.cache/pypoetry/virtualenvs/assembler-01ENdme1-py3.11/lib/python3.11/site-packages/redis/asyncio/connection.py", line 216, in __del__
  File "/home/naltharial/.cache/pypoetry/virtualenvs/assembler-01ENdme1-py3.11/lib/python3.11/site-packages/redis/asyncio/connection.py", line 223, in _close
  File "/usr/lib/python3.11/asyncio/streams.py", line 345, in close
  File "/usr/lib/python3.11/asyncio/selector_events.py", line 831, in close
  File "/usr/lib/python3.11/asyncio/base_events.py", line 758, in call_soon
  File "/usr/lib/python3.11/asyncio/base_events.py", line 519, in _check_closed
RuntimeError: Event loop is closed

it looks like HTTPx is closing the event loop behind it, no matter if Redis is still trying to use it. Is this related to the same issue? Notably, this only occurs under Ubuntu, I can't reproduce it on Windows, so my debugging capacity is limited.

It's resolved (obviously) by using the sync Redis API instead of the async one, but upgrading the libraries didn't work.

Package                       Version
----------------------------- -----------
...
anyio                         4.3.0
httpx                         0.27.0
redis                         5.0.3
trio                          0.25.0
...

I have the same problem. And here is the example

async def main():
    cache = Redis(connection_pool=pool)
    async with cache:
        print(await cache.get("key"))  # it's ok
        async with httpx.AsyncClient() as client:
            print(await cache.get("key"))  # it's ok
            response = await client.get("https://www.github.com")
            print(await cache.get("key"))  # here is broken
            # TODO

PS: It works well for redis-py<5 or anyio<4.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests