Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixing cancelled async futures #2666

Merged
merged 18 commits into from
Mar 29, 2023
Merged

Fixing cancelled async futures #2666

merged 18 commits into from
Mar 29, 2023

Conversation

chayim
Copy link
Contributor

@chayim chayim commented Mar 26, 2023

The following pull request contains changes to Async cancelling. Specifically, all cases where a command is sent to redis, via execute_command (and friends), and now wrapped with a shield. This builds on the helpful proxy provided in #2665.

This change covers the following usage patterns.

  • Async
  • Async pipeline
  • Async pubsub
  • Async sentinel
  • Async cluster
  • Async cluster pipelines

Reproducible tests currently exist for async, async pipeline, and async cluster. But note - the async cluster pipeline seems at best incorrect.

Feedback and changes are very much welcomed. Bonus points for assistance with testing,

chayim and others added 3 commits March 27, 2023 08:47
@codecov-commenter
Copy link

codecov-commenter commented Mar 27, 2023

Codecov Report

Patch coverage: 94.55% and project coverage change: +0.04 🎉

Comparison is base (326bb1c) 92.25% compared to head (bb65a25) 92.29%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2666      +/-   ##
==========================================
+ Coverage   92.25%   92.29%   +0.04%     
==========================================
  Files         115      116       +1     
  Lines       29788    29893     +105     
==========================================
+ Hits        27481    27590     +109     
+ Misses       2307     2303       -4     
Impacted Files Coverage Δ
tests/test_asyncio/test_cluster.py 97.48% <ø> (+0.10%) ⬆️
tests/test_asyncio/test_connection.py 98.06% <ø> (+1.00%) ⬆️
redis/asyncio/cluster.py 91.82% <81.81%> (+0.40%) ⬆️
redis/asyncio/client.py 92.34% <92.10%> (-0.64%) ⬇️
tests/test_asyncio/test_cwe_404.py 96.90% <96.90%> (ø)
tests/test_asyncio/test_pubsub.py 99.06% <100.00%> (-0.16%) ⬇️

... and 3 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

redis/asyncio/client.py Outdated Show resolved Hide resolved
redis/asyncio/client.py Outdated Show resolved Hide resolved
Copy link
Contributor

@sileht sileht left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even in theory, this change should fix the issue. asyncio.CancelledError may never reaches this redispy code due to this bug: aio-libs/async-timeout#229.

@chayim
Copy link
Contributor Author

chayim commented Mar 28, 2023

@fps7806 @sileht I saw that when doing some research on this a couple of days back. That's what lead down some of the paths.

IMHO the right thing to do on the security side - is this series of changes. Yes, this can lead to the side affect of a true connection issue, if the connection has been cancelled. I think, looking at that we can safely say that the use of the client should handle that, as opposed to the client.

We're going to have a look at some try/catch work here. If try/catch within the shield triggers properly - I think we're good for release.

Targeting tomorrow.

@chayim
Copy link
Contributor Author

chayim commented Mar 28, 2023

Relevent: python/cpython#28149

@chayim
Copy link
Contributor Author

chayim commented Mar 28, 2023

Taking a final look at things.. it appears the best way to solve this, it to properly break the pipeline, when this timing issue occurs. In breaking the pipeline only in this case a RuntimeError is raised.

This exception is ultimately raised from below, at least in the specific asyncio/streams code: /usr/lib/python3.10/asyncio/streams.py:616.

As a result, the shield needs to catch the RuntimeError - exception, and we can trigger the pipeline reset. This is necessary because the CancelledError, as alluded to previously, doesn't trigger - though should.

Our choices are either to:

  1. Raise the RuntimeError , and leave it up to the user. This strikes me as inelligant.

  2. Catch it (as done here). In this case the pipeline will reset properly, and return a None.

This final change, has that in it.

@chayim
Copy link
Contributor Author

chayim commented Mar 29, 2023

Failures are the result of #2670 and unrelated.

@chayim chayim merged commit 5acbde3 into redis:master Mar 29, 2023
dvora-h added a commit that referenced this pull request Mar 29, 2023
Co-authored-by: James R T <jamestiotio@gmail.com>
Co-authored-by: dvora-h <dvora.heller@redis.com>
dvora-h added a commit that referenced this pull request Mar 29, 2023
Co-authored-by: James R T <jamestiotio@gmail.com>
Co-authored-by: dvora-h <dvora.heller@redis.com>
dvora-h added a commit that referenced this pull request Mar 29, 2023
* Fixing cancelled async futures (#2666)

Co-authored-by: James R T <jamestiotio@gmail.com>
Co-authored-by: dvora-h <dvora.heller@redis.com>

* Version 4.4.4

* fix  tests

* linters

* fixing the test for the 4.4 state

* remove superfluous try-except

---------

Co-authored-by: Chayim <chayim@users.noreply.github.com>
Co-authored-by: James R T <jamestiotio@gmail.com>
Co-authored-by: Chayim I. Kirshen <c@kirshen.com>
@risicle
Copy link

risicle commented Mar 31, 2023

❤️ for the backports, folks.

(Oh, I see 4.3.7 has been tagged but not released yet)

@kristjanvalur
Copy link
Contributor

This change makes it impossible to cancel a command and retry, because the cancellation never happens and the single connection lock is held indefinetly. So, the unit-tests I devised for pr #2506 no longer work.

Is this really intentional? Not to cancel operation at all? Is there a tl/dr somewhere on why this is desirable?

@kristjanvalur
Copy link
Contributor

Anyone reading the code will be clueless. "Wait, it is creating a new Task for each operation? That adds latency. and then shielding it? Whatever for?" Even the unit tests have no comments on them on what specifically they are trying to fix.

@kristjanvalur
Copy link
Contributor

kristjanvalur commented Apr 4, 2023

This testing code now breaks:

        ready = asyncio.Event()

        async def helper():
            with pytest.raises(asyncio.CancelledError):
                # blocking pop
                ready.set()
                await r.brpop(["nonexist"])
  
          # if all is well, we can continue.  The following should not hang.
    # DEADLOCK, here:
            await r.set("status", "down")

        task = asyncio.create_task(helper())
        await ready.wait()
        await asyncio.sleep(0.01)
        # the task is now sleeping, lets send it an exception
        task.cancel()  

The task was interrupted with a CancelledError. but the inner task (shileded()) was not interrupted and continues to wait, hanging on to the _single_connection_lock and the second attempt at r.set() is now deadlocked.

Basically, task interruption has been nerfed... Is there a workaround?

@agronholm
Copy link
Contributor

For what reason does this shield the sub-task running _try_send_command_parse_response()? If I cancel a task that is performing a blocking command on Redis, the shielded subtask crashes with redis.exceptions.ConnectionError: Connection closed by server. when the client is closed.

In other words: Always manage your tasks! Having unmanaged tasks lying around is bad practice.

@kristjanvalur
Copy link
Contributor

kristjanvalur commented Apr 20, 2023

@agronholm I believe this PR was a mistake. It is an attempt to fix an issue caused by a regression that I caused last year, but without understanding the problem. I have had a fix for that regression lying around for four months without being accepted.

Please see pr #2695 which undoes this change and applies the correct fix to the original regression.

dvora-h added a commit that referenced this pull request Jul 3, 2023
* fix: do not use asyncio's timeout lib before 3.11.2 (#2659)

There's an issue in asyncio's timeout lib before 3.11.3 that causes
async calls to raise `CancelledError`.

This is a cpython issue that was fixed in this commit [1] and
cherry-picked to previous versions, meaning 3.11.3 will work correctly.

Check [2] for more info.

[1] python/cpython@04adf2d
[2] #2633

* UnixDomainSocketConnection missing constructor argument (#2630)

* removing useless files (#2642)

* Fix issue 2660: PytestUnraisableExceptionWarning from asycio client (#2669)

* Fixing cancelled async futures (#2666)

Co-authored-by: James R T <jamestiotio@gmail.com>
Co-authored-by: dvora-h <dvora.heller@redis.com>

* Fix async (#2673)

* Version 4.5.4 (#2674)

* Really do not use asyncio's timeout lib before 3.11.2 (#2699)

4802530 made async-timeout required
only on Python 3.11.2 and earlier. However, according to PEP-508,
python_version marker is compared to first two numbers of Python version
tuple - so it will evaluate to True also on 3.11.3, and install a
package as a dependency.

* asyncio: Fix memory leak caused by hiredis (#2693) (#2694)

* Update example of Redisearch creating index (#2703)

When creating index, fields should be passed inside an iterable (e.g. list or tuple)

* Improving Vector Similarity Search Example (#2661)

* update vss docs

* add embeddings creation and storage examples

* update based on feedback

* fix version and link

* include more realistic search examples and clean up indices

* completely remove initial cap reference

---------

Co-authored-by: Chayim <chayim@users.noreply.github.com>

* Fix incorrect usage of once flag in async Sentinel (#2718)

In the execute_command of the async Sentinel, the once flag was being
used incorrectly, with its meaning inverted. To fix we just needed to invert
the if and else bodies. This isn't being caught by the tests currently
because the tests of commands that use this flag do not check their
results/effects (for example the "test_ckquorum" test).

* Fix topk list example. (#2724)

* Improve error output for master discovery (#2720)

Make MasterNotFoundError exception more precise in the case of
ConnectionError and TimeoutError to help the user to identify
configuration errors

Co-authored-by: Marc Schöchlin <marc.schoechlin@flipapp.de>

* return response in case of KeyError (#2628)

* return response in case of KeyError

* fix code linters error

* fix linters 2

* fix linters 3

* Add WITHSCORES to ZREVRANK Command (#2725)

* add withscores to zrevrank

* change 0 -> 2

* fix errors

* split test

* Fix `ClusterCommandProtocol` not itself being marked as a protocol (#2729)

* Fix `ClusterCommandProtocol` not itself being marked as a protocol

* Update CHANGES

* Fix potential race condition during disconnection (#2719)

When the disconnect() function is called twice in parallel it is possible that
one thread deletes the self._sock reference, while the other thread will
attempt to call .close() on it, leading to an AttributeError.

This situation can routinely be encountered by closing the connection in a
PubSubWorkerThread error handler in a blocking thread (ie. with
sleep_time==None), and then calling .close() on the PubSub object.
The main thread will then run into the disconnect() function, and the listener
thread is woken up by the closure and will race into the disconnect()
function, too.

This can be fixed easily by copying the object reference before doing the
None-check, similar to what we do in the redis.client.close() function.

* add "address_remap" feature to RedisCluster (#2726)

* add cluster "host_port_remap" feature for asyncio.RedisCluster

* Add a unittest for asyncio.RedisCluster

* Add host_port_remap to _sync_ RedisCluster

* add synchronous tests

* rename arg to `address_remap` and take and return an address tuple.

* Add class documentation

* Add CHANGES

* nermina changes from NRedisStack (#2736)

* Updated AWS Elasticache IAM Connection Example (#2702)

Co-authored-by: Nick Gerow <nick.gerow@enlightedinc.com>

* pinning urllib3 to fix CI (#2748)

* Add RedisCluster.remap_host_port, Update tests for CWE 404 (#2706)

* Use provided redis address. Bind to IPv4

* Add missing "await" and perform the correct test for pipe eimpty

* Wait for a send event, rather than rely on sleep time. Excpect cancel errors.

* set delay to 0 except for operation we want to cancel
This speeds up the unit tests considerably by eliminating unnecessary delay.

* Release resources in test

* Fix cluster test to use address_remap and multiple proxies.

* Use context manager to manage DelayProxy

* Mark failing pipeline tests

* lint

* Use a common "master_host" test fixture

* Update redismodules.rst (#2747)

Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>

* Add support for cluster myshardid (#2704)

* feat: adding support for cluster myshardid

* lint fix

* fix: comment fix and async test

* fix: adding version check

* fix lint:

* linters

---------

Co-authored-by: Anuragkillswitch <70265851+Anuragkillswitch@users.noreply.github.com>
Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>
Co-authored-by: dvora-h <dvora.heller@redis.com>

* clean warnings (#2731)

* fix parse_slowlog_get (#2732)

* Optionally disable disconnects in read_response (#2695)

* Add regression tests and fixes for issue #1128

* Fix tests for resumable read_response to use "disconnect_on_error"

* undo prevision fix attempts in async client and cluster

* re-enable cluster test

* Suggestions from code review

* Add CHANGES

* Add client no-touch (#2745)

* Add client no-touch

* Update redis/commands/core.py

Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>

* Update test_commands.py

Improve test_client_no_touch

* Update test_commands.py

Add async version test case

* Chore remove whitespace

Oops

---------

Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>

* fix create single_connection_client from url (#2752)

* Fix `xadd` allow non negative maxlen (#2739)

* Fix xadd allow non negative maxlen

* Update change log

---------

Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>

* Version 4.5.5 (#2753)

* Kristjan/issue #2754: Add missing argument to SentinelManagedConnection.read_response() (#2756)

* Increase timeout for a test which would hang completely if failing.
Timeouts in virtualized CI backends can occasionally fail if too short.

* add "disconnect_on_error" argument to SentinelManagedConnection

* update Changes

* lint

* support JSON.MERGE Command (#2761)

* support JSON.MERGE Command

* linters

* try with abc instead person

* change @skip_ifmodversion_lt to latest ReJSON 2.4.7

* change version

* fix test

* linters

* add async test

* Issue #2749: Remove unnecessary __del__ handlers (#2755)

* Remove unnecessary __del__ handlers
There normally should be no logic attached to del.  Cleanly disconnecting network resources is not needed at that time.

* add CHANGES

* Add WITHSCORE to ZRANK (#2758)

* add withscore to zrank with tests

* fix test

* Fix JSON.MERGE Summary (#2786)

* Fix JSON.MERGE Summary

* linters

* Fixed key error in parse_xinfo_stream (#2788)

* insert newline to prevent sphinx from assuming code block (#2796)

* Introduce OutOfMemoryError exception for Redis write command rejections due to OOM errors (#2778)

* expose OutOfMemoryError as explicit exception type

- handle "OOM" error code string by raising explicit
  exception type instance
- enables callers to avoid string matching after
  catching ResponseError

* add OutOfMemoryError exception class docstring

* Provide more info in the exception docstring

* Fix formatting

* Again

* linters

---------

Co-authored-by: Chayim <chayim@users.noreply.github.com>
Co-authored-by: Igor Malinovskiy <u.glide@gmail.com>
Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>

* Add unit tests for the `connect` method of all Redis connection classes (#2631)

* tests: move certificate discovery to a separate module

* tests: add 'connect' tests for all Redis connection classes

---------

Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>

* Fix dead weakref in sentinel connection causing ReferenceError (#2767) (#2771)

* Fix dead weakref in sentinel conn (#2767)

* Update CHANGES

---------

Co-authored-by: Igor Malinovskiy <u.glide@gmail.com>
Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>

* chore(documentation): fix redirects and some small cleanups (#2801)

* Add waitaof (#2760)

* Add waitaof

* Update test_commands.py

add test_waitaof

* Update test_commands.py

Add test_waitaof

* Fix doc string

---------

Co-authored-by: Chayim <chayim@users.noreply.github.com>
Co-authored-by: Igor Malinovskiy <u.glide@gmail.com>

* Extract abstract async connection class (#2734)

* make 'socket_timeout' and 'socket_connect_timeout' equivalent for TCP and UDS connections

* abstract asynio connection in analogy with the synchronous connection

---------

Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>

* Fix type hint for retry_on_error in async cluster (#2804)

* fix(asyncio.cluster): fixup retry_on_error type hint

This parameter accepts a list of _classes of Exceptions_, not a list of instantiated Exceptions. Fixup the type hint accordingly.

* chore: update changelog

---------

Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>

* Fix CI (#2809)

* Support JSON.MSET Command (#2766)

* support JSON.MERGE Command

* linters

* try with abc instead person

* change @skip_ifmodversion_lt to latest ReJSON 2.4.7

* change version

* fix test

* linters

* add async test

* Support JSON.MSET command

* trying to run CI

* linters

* add async test

* reminder do delete the integration changes

* delete the line from integration

* fix the interface

* change docstring

---------

Co-authored-by: Chayim <chayim@users.noreply.github.com>
Co-authored-by: dvora-h <dvora.heller@redis.com>

* Version 4.6.0 (#2810)

* master changes

* linters

* fix test_cwe_404 cluster test

---------

Co-authored-by: Thiago Bellini Ribeiro <hackedbellini@gmail.com>
Co-authored-by: woutdenolf <woutdenolf@users.sf.net>
Co-authored-by: Chayim <chayim@users.noreply.github.com>
Co-authored-by: shacharPash <93581407+shacharPash@users.noreply.github.com>
Co-authored-by: James R T <jamestiotio@gmail.com>
Co-authored-by: Mirek Długosz <miniopl+github@gmail.com>
Co-authored-by: Oran Avraham <252748+oranav@users.noreply.github.com>
Co-authored-by: mzdehbashi-github <85902780+mzdehbashi-github@users.noreply.github.com>
Co-authored-by: Tyler Hutcherson <tyler.hutcherson@redis.com>
Co-authored-by: Felipe Machado <462154+felipou@users.noreply.github.com>
Co-authored-by: AYMEN Mohammed <53928879+AYMENJD@users.noreply.github.com>
Co-authored-by: Marc Schöchlin <ms-github@256bit.org>
Co-authored-by: Marc Schöchlin <marc.schoechlin@flipapp.de>
Co-authored-by: Avasam <samuel.06@hotmail.com>
Co-authored-by: Markus Gerstel <2102431+Anthchirp@users.noreply.github.com>
Co-authored-by: Kristján Valur Jónsson <sweskman@gmail.com>
Co-authored-by: Nick Gerow <Nick.G.123@hotmail.com>
Co-authored-by: Nick Gerow <nick.gerow@enlightedinc.com>
Co-authored-by: Cristian Matache <cristianmatache@hotmail.com>
Co-authored-by: Anurag Bandyopadhyay <angbpy@gmail.com>
Co-authored-by: Anuragkillswitch <70265851+Anuragkillswitch@users.noreply.github.com>
Co-authored-by: Seongchuel Ahn <aciddust20@gmail.com>
Co-authored-by: Alibi <aliby.bbb@gmail.com>
Co-authored-by: Smit Parmar <smitraj333@gmail.com>
Co-authored-by: Brad MacPhee <macphee@gmail.com>
Co-authored-by: Igor Malinovskiy <u.glide@gmail.com>
Co-authored-by: Shahar Lev <shahar_lev@hotmail.com>
Co-authored-by: Vladimir Mihailenco <vladimir.webdev@gmail.com>
Co-authored-by: Kevin James <KevinJames@thekev.in>
dvora-h added a commit that referenced this pull request Jul 16, 2023
* Reorganizing the parsers code, and add support for RESP3 (#2574)

* Reorganizing the parsers code

* fix build package

* fix imports

* fix flake8

* add resp to Connection class

* core commands

* python resp3 parser

* pipeline

* async resp3 parser

* some asymc tests

* resp3 parser for async cluster

* async commands tests

* linters

* linters

* linters

* fix ModuleNotFoundError

* fix tests

* fix assert_resp_response_in

* fix command_getkeys in cluster

* fail-fast false

* version

---------

Co-authored-by: Chayim I. Kirshen <c@kirshen.com>

* Fix async client with resp3 (#2657)

* Add support for PubSub with RESP3 parser (#2721)

* add resp3 pubsub

* linters

* _set_info_logger func

* async pubsun

* docstring

* 5.0.0b2 (#2723)

* Fix `COMMAND` response in resp3 (redis 7+) (#2740)

* Fix protocol version checking (#2737)

* bumping beta version to 5.0.0b3 (#2743)

* Fix parse resp3 dict response: don't use dict comprehension (#2757)

* Fix parse respp3 dict response

* linters

* pin urlib version

* Sharded pubsub (#2762)

* sharded pubsub

* sharded pubsub

Co-authored-by: Leibale Eidelman <me@leibale.com>

* Shrded Pubsub TestPubSubSubscribeUnsubscribe

* fix TestPubSubSubscribeUnsubscribe

* more tests

* linters

* TestPubSubSubcommands

* fix @leibale comments

* linters

* fix @chayim comments

---------

Co-authored-by: Leibale Eidelman <me@leibale.com>

* 5.0.0b4 (#2781)

Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>

* RESP3 tests (#2780)

* fix command response in resp3

* linters

* acl_log & acl_getuser

* client_info

* test_commands and test_asyncio/test_commands

* fix test_command_parser

* fix asyncio/test_connection/test_invalid_response

* linters

* all the tests

* push handler sharded pubsub

* Use assert_resp_response wherever possible

* fix test_xreadgroup

* fix cluster_zdiffstore and cluster_zinter

* fix review comments

* fix review comments

* linters

* Fixing asyncio import (#2759)

* asyncio import fix

* pinning urllib3 to fix CI (#2748)

* noqa

* fixint linters

* fix (#2799)

* RESP3 response callbacks (#2798)

* start cleaning

* clean sone callbacks

* response callbacks

* revert redismod-url change

* fix async tests

* linters

* async cluster

---------

Co-authored-by: Chayim <chayim@users.noreply.github.com>

* RESP3 modules support (#2803)

* start cleaning

* clean sone callbacks

* response callbacks

* modules

* tests

* finish sync search tests

* linters

* async modules

* linters

* revert redismod-url change

* RESP3 fix async tests (#2806)

* fix tests

* add stralgo callback in resp2

* add callback to acl list in resp2

* Adding RESP3 tests support (#2793)

* start cleaning

* clean sone callbacks

* first phase

* tox wrap back

* changing cancel format

* syntax

* lint

* docker

* contain the docker

* tox dev reqs

* back to testing

* response callbacks

* protocol into async conftest

* fix for 3.11 invoke

* docker changes

* fix tests

* linters

* adding

* resp3 tox, until killed

* remove tox

* tests

* requirements.txt

* restoring requirements.txt

* adding a sleep, hopefully enough time for the cluster dockers to settle

* fix search tests

* search test, disable uvloop for pypy due to bug

* syn

* reg

* dialect test improvement

* sleep+, xfail

* tests

* resp

* flaky search test too

* timing

* timing for async test

* test changes

* fix assert_interval_advanced

* revert

* mark async health_check tests with xfail

* change strict to false

* fix github actions package validation

---------

Co-authored-by: dvora-h <dvora.heller@redis.com>

* change sismember return type (#2813)

* Version 5.0.0rc1 (#2815)

* Merge master to 5.0 (#2827)

* fix: do not use asyncio's timeout lib before 3.11.2 (#2659)

There's an issue in asyncio's timeout lib before 3.11.3 that causes
async calls to raise `CancelledError`.

This is a cpython issue that was fixed in this commit [1] and
cherry-picked to previous versions, meaning 3.11.3 will work correctly.

Check [2] for more info.

[1] python/cpython@04adf2d
[2] #2633

* UnixDomainSocketConnection missing constructor argument (#2630)

* removing useless files (#2642)

* Fix issue 2660: PytestUnraisableExceptionWarning from asycio client (#2669)

* Fixing cancelled async futures (#2666)

Co-authored-by: James R T <jamestiotio@gmail.com>
Co-authored-by: dvora-h <dvora.heller@redis.com>

* Fix async (#2673)

* Version 4.5.4 (#2674)

* Really do not use asyncio's timeout lib before 3.11.2 (#2699)

4802530 made async-timeout required
only on Python 3.11.2 and earlier. However, according to PEP-508,
python_version marker is compared to first two numbers of Python version
tuple - so it will evaluate to True also on 3.11.3, and install a
package as a dependency.

* asyncio: Fix memory leak caused by hiredis (#2693) (#2694)

* Update example of Redisearch creating index (#2703)

When creating index, fields should be passed inside an iterable (e.g. list or tuple)

* Improving Vector Similarity Search Example (#2661)

* update vss docs

* add embeddings creation and storage examples

* update based on feedback

* fix version and link

* include more realistic search examples and clean up indices

* completely remove initial cap reference

---------

Co-authored-by: Chayim <chayim@users.noreply.github.com>

* Fix incorrect usage of once flag in async Sentinel (#2718)

In the execute_command of the async Sentinel, the once flag was being
used incorrectly, with its meaning inverted. To fix we just needed to invert
the if and else bodies. This isn't being caught by the tests currently
because the tests of commands that use this flag do not check their
results/effects (for example the "test_ckquorum" test).

* Fix topk list example. (#2724)

* Improve error output for master discovery (#2720)

Make MasterNotFoundError exception more precise in the case of
ConnectionError and TimeoutError to help the user to identify
configuration errors

Co-authored-by: Marc Schöchlin <marc.schoechlin@flipapp.de>

* return response in case of KeyError (#2628)

* return response in case of KeyError

* fix code linters error

* fix linters 2

* fix linters 3

* Add WITHSCORES to ZREVRANK Command (#2725)

* add withscores to zrevrank

* change 0 -> 2

* fix errors

* split test

* Fix `ClusterCommandProtocol` not itself being marked as a protocol (#2729)

* Fix `ClusterCommandProtocol` not itself being marked as a protocol

* Update CHANGES

* Fix potential race condition during disconnection (#2719)

When the disconnect() function is called twice in parallel it is possible that
one thread deletes the self._sock reference, while the other thread will
attempt to call .close() on it, leading to an AttributeError.

This situation can routinely be encountered by closing the connection in a
PubSubWorkerThread error handler in a blocking thread (ie. with
sleep_time==None), and then calling .close() on the PubSub object.
The main thread will then run into the disconnect() function, and the listener
thread is woken up by the closure and will race into the disconnect()
function, too.

This can be fixed easily by copying the object reference before doing the
None-check, similar to what we do in the redis.client.close() function.

* add "address_remap" feature to RedisCluster (#2726)

* add cluster "host_port_remap" feature for asyncio.RedisCluster

* Add a unittest for asyncio.RedisCluster

* Add host_port_remap to _sync_ RedisCluster

* add synchronous tests

* rename arg to `address_remap` and take and return an address tuple.

* Add class documentation

* Add CHANGES

* nermina changes from NRedisStack (#2736)

* Updated AWS Elasticache IAM Connection Example (#2702)

Co-authored-by: Nick Gerow <nick.gerow@enlightedinc.com>

* pinning urllib3 to fix CI (#2748)

* Add RedisCluster.remap_host_port, Update tests for CWE 404 (#2706)

* Use provided redis address. Bind to IPv4

* Add missing "await" and perform the correct test for pipe eimpty

* Wait for a send event, rather than rely on sleep time. Excpect cancel errors.

* set delay to 0 except for operation we want to cancel
This speeds up the unit tests considerably by eliminating unnecessary delay.

* Release resources in test

* Fix cluster test to use address_remap and multiple proxies.

* Use context manager to manage DelayProxy

* Mark failing pipeline tests

* lint

* Use a common "master_host" test fixture

* Update redismodules.rst (#2747)

Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>

* Add support for cluster myshardid (#2704)

* feat: adding support for cluster myshardid

* lint fix

* fix: comment fix and async test

* fix: adding version check

* fix lint:

* linters

---------

Co-authored-by: Anuragkillswitch <70265851+Anuragkillswitch@users.noreply.github.com>
Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>
Co-authored-by: dvora-h <dvora.heller@redis.com>

* clean warnings (#2731)

* fix parse_slowlog_get (#2732)

* Optionally disable disconnects in read_response (#2695)

* Add regression tests and fixes for issue #1128

* Fix tests for resumable read_response to use "disconnect_on_error"

* undo prevision fix attempts in async client and cluster

* re-enable cluster test

* Suggestions from code review

* Add CHANGES

* Add client no-touch (#2745)

* Add client no-touch

* Update redis/commands/core.py

Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>

* Update test_commands.py

Improve test_client_no_touch

* Update test_commands.py

Add async version test case

* Chore remove whitespace

Oops

---------

Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>

* fix create single_connection_client from url (#2752)

* Fix `xadd` allow non negative maxlen (#2739)

* Fix xadd allow non negative maxlen

* Update change log

---------

Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>

* Version 4.5.5 (#2753)

* Kristjan/issue #2754: Add missing argument to SentinelManagedConnection.read_response() (#2756)

* Increase timeout for a test which would hang completely if failing.
Timeouts in virtualized CI backends can occasionally fail if too short.

* add "disconnect_on_error" argument to SentinelManagedConnection

* update Changes

* lint

* support JSON.MERGE Command (#2761)

* support JSON.MERGE Command

* linters

* try with abc instead person

* change @skip_ifmodversion_lt to latest ReJSON 2.4.7

* change version

* fix test

* linters

* add async test

* Issue #2749: Remove unnecessary __del__ handlers (#2755)

* Remove unnecessary __del__ handlers
There normally should be no logic attached to del.  Cleanly disconnecting network resources is not needed at that time.

* add CHANGES

* Add WITHSCORE to ZRANK (#2758)

* add withscore to zrank with tests

* fix test

* Fix JSON.MERGE Summary (#2786)

* Fix JSON.MERGE Summary

* linters

* Fixed key error in parse_xinfo_stream (#2788)

* insert newline to prevent sphinx from assuming code block (#2796)

* Introduce OutOfMemoryError exception for Redis write command rejections due to OOM errors (#2778)

* expose OutOfMemoryError as explicit exception type

- handle "OOM" error code string by raising explicit
  exception type instance
- enables callers to avoid string matching after
  catching ResponseError

* add OutOfMemoryError exception class docstring

* Provide more info in the exception docstring

* Fix formatting

* Again

* linters

---------

Co-authored-by: Chayim <chayim@users.noreply.github.com>
Co-authored-by: Igor Malinovskiy <u.glide@gmail.com>
Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>

* Add unit tests for the `connect` method of all Redis connection classes (#2631)

* tests: move certificate discovery to a separate module

* tests: add 'connect' tests for all Redis connection classes

---------

Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>

* Fix dead weakref in sentinel connection causing ReferenceError (#2767) (#2771)

* Fix dead weakref in sentinel conn (#2767)

* Update CHANGES

---------

Co-authored-by: Igor Malinovskiy <u.glide@gmail.com>
Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>

* chore(documentation): fix redirects and some small cleanups (#2801)

* Add waitaof (#2760)

* Add waitaof

* Update test_commands.py

add test_waitaof

* Update test_commands.py

Add test_waitaof

* Fix doc string

---------

Co-authored-by: Chayim <chayim@users.noreply.github.com>
Co-authored-by: Igor Malinovskiy <u.glide@gmail.com>

* Extract abstract async connection class (#2734)

* make 'socket_timeout' and 'socket_connect_timeout' equivalent for TCP and UDS connections

* abstract asynio connection in analogy with the synchronous connection

---------

Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>

* Fix type hint for retry_on_error in async cluster (#2804)

* fix(asyncio.cluster): fixup retry_on_error type hint

This parameter accepts a list of _classes of Exceptions_, not a list of instantiated Exceptions. Fixup the type hint accordingly.

* chore: update changelog

---------

Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>

* Fix CI (#2809)

* Support JSON.MSET Command (#2766)

* support JSON.MERGE Command

* linters

* try with abc instead person

* change @skip_ifmodversion_lt to latest ReJSON 2.4.7

* change version

* fix test

* linters

* add async test

* Support JSON.MSET command

* trying to run CI

* linters

* add async test

* reminder do delete the integration changes

* delete the line from integration

* fix the interface

* change docstring

---------

Co-authored-by: Chayim <chayim@users.noreply.github.com>
Co-authored-by: dvora-h <dvora.heller@redis.com>

* Version 4.6.0 (#2810)

* master changes

* linters

* fix test_cwe_404 cluster test

---------

Co-authored-by: Thiago Bellini Ribeiro <hackedbellini@gmail.com>
Co-authored-by: woutdenolf <woutdenolf@users.sf.net>
Co-authored-by: Chayim <chayim@users.noreply.github.com>
Co-authored-by: shacharPash <93581407+shacharPash@users.noreply.github.com>
Co-authored-by: James R T <jamestiotio@gmail.com>
Co-authored-by: Mirek Długosz <miniopl+github@gmail.com>
Co-authored-by: Oran Avraham <252748+oranav@users.noreply.github.com>
Co-authored-by: mzdehbashi-github <85902780+mzdehbashi-github@users.noreply.github.com>
Co-authored-by: Tyler Hutcherson <tyler.hutcherson@redis.com>
Co-authored-by: Felipe Machado <462154+felipou@users.noreply.github.com>
Co-authored-by: AYMEN Mohammed <53928879+AYMENJD@users.noreply.github.com>
Co-authored-by: Marc Schöchlin <ms-github@256bit.org>
Co-authored-by: Marc Schöchlin <marc.schoechlin@flipapp.de>
Co-authored-by: Avasam <samuel.06@hotmail.com>
Co-authored-by: Markus Gerstel <2102431+Anthchirp@users.noreply.github.com>
Co-authored-by: Kristján Valur Jónsson <sweskman@gmail.com>
Co-authored-by: Nick Gerow <Nick.G.123@hotmail.com>
Co-authored-by: Nick Gerow <nick.gerow@enlightedinc.com>
Co-authored-by: Cristian Matache <cristianmatache@hotmail.com>
Co-authored-by: Anurag Bandyopadhyay <angbpy@gmail.com>
Co-authored-by: Anuragkillswitch <70265851+Anuragkillswitch@users.noreply.github.com>
Co-authored-by: Seongchuel Ahn <aciddust20@gmail.com>
Co-authored-by: Alibi <aliby.bbb@gmail.com>
Co-authored-by: Smit Parmar <smitraj333@gmail.com>
Co-authored-by: Brad MacPhee <macphee@gmail.com>
Co-authored-by: Igor Malinovskiy <u.glide@gmail.com>
Co-authored-by: Shahar Lev <shahar_lev@hotmail.com>
Co-authored-by: Vladimir Mihailenco <vladimir.webdev@gmail.com>
Co-authored-by: Kevin James <KevinJames@thekev.in>

* RESP3 response-callbacks cleanup (#2841)

* cluenup

* sentinel callbacks

* move callbacks

* fix async cluster tests

* _parsers and import fix in tests

* linters

* make modules callbacks private

* fix async search

* fix

---------

Co-authored-by: Chayim I. Kirshen <c@kirshen.com>

* Version 5.0.0rc2 (#2843)

* linters

---------

Co-authored-by: Chayim I. Kirshen <c@kirshen.com>
Co-authored-by: Chayim <chayim@users.noreply.github.com>
Co-authored-by: Leibale Eidelman <me@leibale.com>
Co-authored-by: Thiago Bellini Ribeiro <hackedbellini@gmail.com>
Co-authored-by: woutdenolf <woutdenolf@users.sf.net>
Co-authored-by: shacharPash <93581407+shacharPash@users.noreply.github.com>
Co-authored-by: James R T <jamestiotio@gmail.com>
Co-authored-by: Mirek Długosz <miniopl+github@gmail.com>
Co-authored-by: Oran Avraham <252748+oranav@users.noreply.github.com>
Co-authored-by: mzdehbashi-github <85902780+mzdehbashi-github@users.noreply.github.com>
Co-authored-by: Tyler Hutcherson <tyler.hutcherson@redis.com>
Co-authored-by: Felipe Machado <462154+felipou@users.noreply.github.com>
Co-authored-by: AYMEN Mohammed <53928879+AYMENJD@users.noreply.github.com>
Co-authored-by: Marc Schöchlin <ms-github@256bit.org>
Co-authored-by: Marc Schöchlin <marc.schoechlin@flipapp.de>
Co-authored-by: Avasam <samuel.06@hotmail.com>
Co-authored-by: Markus Gerstel <2102431+Anthchirp@users.noreply.github.com>
Co-authored-by: Kristján Valur Jónsson <sweskman@gmail.com>
Co-authored-by: Nick Gerow <Nick.G.123@hotmail.com>
Co-authored-by: Nick Gerow <nick.gerow@enlightedinc.com>
Co-authored-by: Cristian Matache <cristianmatache@hotmail.com>
Co-authored-by: Anurag Bandyopadhyay <angbpy@gmail.com>
Co-authored-by: Anuragkillswitch <70265851+Anuragkillswitch@users.noreply.github.com>
Co-authored-by: Seongchuel Ahn <aciddust20@gmail.com>
Co-authored-by: Alibi <aliby.bbb@gmail.com>
Co-authored-by: Smit Parmar <smitraj333@gmail.com>
Co-authored-by: Brad MacPhee <macphee@gmail.com>
Co-authored-by: Igor Malinovskiy <u.glide@gmail.com>
Co-authored-by: Shahar Lev <shahar_lev@hotmail.com>
Co-authored-by: Vladimir Mihailenco <vladimir.webdev@gmail.com>
Co-authored-by: Kevin James <KevinJames@thekev.in>
chayim pushed a commit that referenced this pull request Aug 31, 2023
Co-authored-by: Viktor Ivanov <viktor@infogrid.io>
Co-authored-by: Sergey Prokazov <sergey.prokazov@redis.com>
Co-authored-by: Anuragkillswitch <70265851+Anuragkillswitch@users.noreply.github.com>
Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>
Co-authored-by: Alex Schmitz <aschmitz@box.com>
Co-authored-by: Alex Schmitz <alex.schmitz@gmail.com>
Co-authored-by: Chayim <chayim@users.noreply.github.com>
Co-authored-by: Bar Shaul <88437685+barshaul@users.noreply.github.com>
Co-authored-by: CrimsonGlory <CrimsonGlory@users.noreply.github.com>
Co-authored-by: Raymond Yin <raymond@tryevergreen.com>
Co-authored-by: zach.lee <zach.lee@sendbird.com>
Co-authored-by: James R T <jamestiotio@gmail.com>
Co-authored-by: dvora-h <dvora.heller@redis.com>
Co-authored-by: Marc Schöchlin <marc.schoechlin@flipapp.de>
Co-authored-by: Nick Gerow <nick.gerow@enlightedinc.com>
Co-authored-by: Igor Malinovskiy <u.glide@gmail.com>
Co-authored-by: Chayim I. Kirshen <c@kirshen.com>
Co-authored-by: Leibale Eidelman <me@leibale.com>
Co-authored-by: Thiago Bellini Ribeiro <hackedbellini@gmail.com>
Co-authored-by: woutdenolf <woutdenolf@users.sf.net>
Co-authored-by: shacharPash <93581407+shacharPash@users.noreply.github.com>
Co-authored-by: Mirek Długosz <miniopl+github@gmail.com>
Co-authored-by: Oran Avraham <252748+oranav@users.noreply.github.com>
Co-authored-by: mzdehbashi-github <85902780+mzdehbashi-github@users.noreply.github.com>
Co-authored-by: Tyler Hutcherson <tyler.hutcherson@redis.com>
Co-authored-by: Felipe Machado <462154+felipou@users.noreply.github.com>
Co-authored-by: AYMEN Mohammed <53928879+AYMENJD@users.noreply.github.com>
Co-authored-by: Marc Schöchlin <ms-github@256bit.org>
Co-authored-by: Avasam <samuel.06@hotmail.com>
Co-authored-by: Markus Gerstel <2102431+Anthchirp@users.noreply.github.com>
Co-authored-by: Kristján Valur Jónsson <sweskman@gmail.com>
Co-authored-by: Nick Gerow <Nick.G.123@hotmail.com>
Co-authored-by: Cristian Matache <cristianmatache@hotmail.com>
Co-authored-by: Anurag Bandyopadhyay <angbpy@gmail.com>
Co-authored-by: Seongchuel Ahn <aciddust20@gmail.com>
Co-authored-by: Alibi <aliby.bbb@gmail.com>
Co-authored-by: Smit Parmar <smitraj333@gmail.com>
Co-authored-by: Brad MacPhee <macphee@gmail.com>
Co-authored-by: Shahar Lev <shahar_lev@hotmail.com>
Co-authored-by: Vladimir Mihailenco <vladimir.webdev@gmail.com>
Co-authored-by: Kevin James <KevinJames@thekev.in>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: David Pacsuta <34983281+ant1fact@users.noreply.github.com>
Co-authored-by: Rich Bowen <rbowen@rcbowen.com>
Co-authored-by: gmbnomis <gmbnomis@users.noreply.github.com>
Co-authored-by: Vivanov98 <66319645+Vivanov98@users.noreply.github.com>
Co-authored-by: Kosuke <kosuke.zhang@gmail.com>
Co-authored-by: Sergey Prokazov <prokazov@users.noreply.github.com>
Co-authored-by: jmcbailey <jmcbailey@gmail.com>
Co-authored-by: Galtozzy <14139502+Galtozzy@users.noreply.github.com>
Co-authored-by: Abhishek Kumar Sinha <aksinha334@gmail.com>
Co-authored-by: Eom Taegyung "Iggy <iggy.eom@sendbird.com>
Co-authored-by: Mehdi ABAAKOUK <sileht@sileht.net>
Co-authored-by: Dongkeun Lee <3315213+zakaf@users.noreply.github.com>
Co-authored-by: woutdenolf <wout.de_nolf@esrf.eu>
Co-authored-by: Kurt McKee <contactme@kurtmckee.org>
Co-authored-by: Juraj Páll <palljuraj1@gmail.com>
Co-authored-by: Joan Fontanals <jfontanalsmartinez@gmail.com>
Co-authored-by: Stanislav Zmiev <zmievsa@gmail.com>
fix (#2566)
Fix unlink in cluster pipeline (#2562)
Fix issue 2540: Synchronise concurrent command calls to single-client mode. (#2568)
Fix: tuple function cannot be passed more than one argument (#2573)
Fix issue 2567: NoneType check before raising exception (#2569)
Fix issue 2349: Let async HiredisParser finish parsing after a Connection.disconnect() (#2557)
Fix issue with `pack_commands` returning an empty byte sequence (#2416)
Fix #2581 UnixDomainSocketConnection' object has no attribute '_command_packer' (#2583)
Fix #2581 UnixDomainSocketConnection' object has no attribute '_command_packer' .
Fix for `lpop` and `rpop` return typing (#2590)
Fixed CredentialsProvider examples (#2587)
Fixed issue #2598 - make Document class subscriptable
fix: replace async_timeout by asyncio.timeout (#2602)
Fix behaviour of async PythonParser to match RedisParser as for issue #2349 (#2582)
Fix (#2641)
fix: do not use asyncio's timeout lib before 3.11.2 (#2659)
Fix issue 2660: PytestUnraisableExceptionWarning from asycio client (#2669)
Fixing cancelled async futures (#2666)
Fix async (#2673)
Fix memory leak caused by hiredis (#2693) (#2694)
Fix incorrect usage of once flag in async Sentinel (#2718)
Fix topk list example. (#2724)
Fix `ClusterCommandProtocol` not itself being marked as a protocol (#2729)
Fix potential race condition during disconnection (#2719)
fix CI (#2748)
fix parse_slowlog_get (#2732)
fixes for issue #1128
fix create single_connection_client from url (#2752)
Fix `xadd` allow non negative maxlen (#2739)
Fix JSON.MERGE Summary (#2786)
Fixed key error in parse_xinfo_stream (#2788)
Fix dead weakref in sentinel connection causing ReferenceError (#2767) (#2771)
Fix dead weakref in sentinel conn (#2767)
fix redirects and some small cleanups (#2801)
Fix type hint for retry_on_error in async cluster (#2804)
Fix CI (#2809)
Fix async client with resp3 (#2657)
Fix `COMMAND` response in resp3 (redis 7+) (#2740)
Fix protocol version checking (#2737)
Fix parse resp3 dict response: don't use dict comprehension (#2757)
Fixing asyncio import (#2759)
fix (#2799)
fix async tests (#2806)
Fix socket garbage collection (#2859)
Fixing doc builds (#2869)
Fix a duplicate word in `CONTRIBUTING.md` (#2848)
Fix timeout retrying on Redis pipeline execution (#2812)
Fix type hints in SearchCommands (#2817)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants