Skip to content

Commit

Permalink
Merge master to 5.0 (#2827)
Browse files Browse the repository at this point in the history
* fix: do not use asyncio's timeout lib before 3.11.2 (#2659)

There's an issue in asyncio's timeout lib before 3.11.3 that causes
async calls to raise `CancelledError`.

This is a cpython issue that was fixed in this commit [1] and
cherry-picked to previous versions, meaning 3.11.3 will work correctly.

Check [2] for more info.

[1] python/cpython@04adf2d
[2] #2633

* UnixDomainSocketConnection missing constructor argument (#2630)

* removing useless files (#2642)

* Fix issue 2660: PytestUnraisableExceptionWarning from asycio client (#2669)

* Fixing cancelled async futures (#2666)

Co-authored-by: James R T <jamestiotio@gmail.com>
Co-authored-by: dvora-h <dvora.heller@redis.com>

* Fix async (#2673)

* Version 4.5.4 (#2674)

* Really do not use asyncio's timeout lib before 3.11.2 (#2699)

4802530 made async-timeout required
only on Python 3.11.2 and earlier. However, according to PEP-508,
python_version marker is compared to first two numbers of Python version
tuple - so it will evaluate to True also on 3.11.3, and install a
package as a dependency.

* asyncio: Fix memory leak caused by hiredis (#2693) (#2694)

* Update example of Redisearch creating index (#2703)

When creating index, fields should be passed inside an iterable (e.g. list or tuple)

* Improving Vector Similarity Search Example (#2661)

* update vss docs

* add embeddings creation and storage examples

* update based on feedback

* fix version and link

* include more realistic search examples and clean up indices

* completely remove initial cap reference

---------

Co-authored-by: Chayim <chayim@users.noreply.github.com>

* Fix incorrect usage of once flag in async Sentinel (#2718)

In the execute_command of the async Sentinel, the once flag was being
used incorrectly, with its meaning inverted. To fix we just needed to invert
the if and else bodies. This isn't being caught by the tests currently
because the tests of commands that use this flag do not check their
results/effects (for example the "test_ckquorum" test).

* Fix topk list example. (#2724)

* Improve error output for master discovery (#2720)

Make MasterNotFoundError exception more precise in the case of
ConnectionError and TimeoutError to help the user to identify
configuration errors

Co-authored-by: Marc Schöchlin <marc.schoechlin@flipapp.de>

* return response in case of KeyError (#2628)

* return response in case of KeyError

* fix code linters error

* fix linters 2

* fix linters 3

* Add WITHSCORES to ZREVRANK Command (#2725)

* add withscores to zrevrank

* change 0 -> 2

* fix errors

* split test

* Fix `ClusterCommandProtocol` not itself being marked as a protocol (#2729)

* Fix `ClusterCommandProtocol` not itself being marked as a protocol

* Update CHANGES

* Fix potential race condition during disconnection (#2719)

When the disconnect() function is called twice in parallel it is possible that
one thread deletes the self._sock reference, while the other thread will
attempt to call .close() on it, leading to an AttributeError.

This situation can routinely be encountered by closing the connection in a
PubSubWorkerThread error handler in a blocking thread (ie. with
sleep_time==None), and then calling .close() on the PubSub object.
The main thread will then run into the disconnect() function, and the listener
thread is woken up by the closure and will race into the disconnect()
function, too.

This can be fixed easily by copying the object reference before doing the
None-check, similar to what we do in the redis.client.close() function.

* add "address_remap" feature to RedisCluster (#2726)

* add cluster "host_port_remap" feature for asyncio.RedisCluster

* Add a unittest for asyncio.RedisCluster

* Add host_port_remap to _sync_ RedisCluster

* add synchronous tests

* rename arg to `address_remap` and take and return an address tuple.

* Add class documentation

* Add CHANGES

* nermina changes from NRedisStack (#2736)

* Updated AWS Elasticache IAM Connection Example (#2702)

Co-authored-by: Nick Gerow <nick.gerow@enlightedinc.com>

* pinning urllib3 to fix CI (#2748)

* Add RedisCluster.remap_host_port, Update tests for CWE 404 (#2706)

* Use provided redis address. Bind to IPv4

* Add missing "await" and perform the correct test for pipe eimpty

* Wait for a send event, rather than rely on sleep time. Excpect cancel errors.

* set delay to 0 except for operation we want to cancel
This speeds up the unit tests considerably by eliminating unnecessary delay.

* Release resources in test

* Fix cluster test to use address_remap and multiple proxies.

* Use context manager to manage DelayProxy

* Mark failing pipeline tests

* lint

* Use a common "master_host" test fixture

* Update redismodules.rst (#2747)

Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>

* Add support for cluster myshardid (#2704)

* feat: adding support for cluster myshardid

* lint fix

* fix: comment fix and async test

* fix: adding version check

* fix lint:

* linters

---------

Co-authored-by: Anuragkillswitch <70265851+Anuragkillswitch@users.noreply.github.com>
Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>
Co-authored-by: dvora-h <dvora.heller@redis.com>

* clean warnings (#2731)

* fix parse_slowlog_get (#2732)

* Optionally disable disconnects in read_response (#2695)

* Add regression tests and fixes for issue #1128

* Fix tests for resumable read_response to use "disconnect_on_error"

* undo prevision fix attempts in async client and cluster

* re-enable cluster test

* Suggestions from code review

* Add CHANGES

* Add client no-touch (#2745)

* Add client no-touch

* Update redis/commands/core.py

Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>

* Update test_commands.py

Improve test_client_no_touch

* Update test_commands.py

Add async version test case

* Chore remove whitespace

Oops

---------

Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>

* fix create single_connection_client from url (#2752)

* Fix `xadd` allow non negative maxlen (#2739)

* Fix xadd allow non negative maxlen

* Update change log

---------

Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>

* Version 4.5.5 (#2753)

* Kristjan/issue #2754: Add missing argument to SentinelManagedConnection.read_response() (#2756)

* Increase timeout for a test which would hang completely if failing.
Timeouts in virtualized CI backends can occasionally fail if too short.

* add "disconnect_on_error" argument to SentinelManagedConnection

* update Changes

* lint

* support JSON.MERGE Command (#2761)

* support JSON.MERGE Command

* linters

* try with abc instead person

* change @skip_ifmodversion_lt to latest ReJSON 2.4.7

* change version

* fix test

* linters

* add async test

* Issue #2749: Remove unnecessary __del__ handlers (#2755)

* Remove unnecessary __del__ handlers
There normally should be no logic attached to del.  Cleanly disconnecting network resources is not needed at that time.

* add CHANGES

* Add WITHSCORE to ZRANK (#2758)

* add withscore to zrank with tests

* fix test

* Fix JSON.MERGE Summary (#2786)

* Fix JSON.MERGE Summary

* linters

* Fixed key error in parse_xinfo_stream (#2788)

* insert newline to prevent sphinx from assuming code block (#2796)

* Introduce OutOfMemoryError exception for Redis write command rejections due to OOM errors (#2778)

* expose OutOfMemoryError as explicit exception type

- handle "OOM" error code string by raising explicit
  exception type instance
- enables callers to avoid string matching after
  catching ResponseError

* add OutOfMemoryError exception class docstring

* Provide more info in the exception docstring

* Fix formatting

* Again

* linters

---------

Co-authored-by: Chayim <chayim@users.noreply.github.com>
Co-authored-by: Igor Malinovskiy <u.glide@gmail.com>
Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>

* Add unit tests for the `connect` method of all Redis connection classes (#2631)

* tests: move certificate discovery to a separate module

* tests: add 'connect' tests for all Redis connection classes

---------

Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>

* Fix dead weakref in sentinel connection causing ReferenceError (#2767) (#2771)

* Fix dead weakref in sentinel conn (#2767)

* Update CHANGES

---------

Co-authored-by: Igor Malinovskiy <u.glide@gmail.com>
Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>

* chore(documentation): fix redirects and some small cleanups (#2801)

* Add waitaof (#2760)

* Add waitaof

* Update test_commands.py

add test_waitaof

* Update test_commands.py

Add test_waitaof

* Fix doc string

---------

Co-authored-by: Chayim <chayim@users.noreply.github.com>
Co-authored-by: Igor Malinovskiy <u.glide@gmail.com>

* Extract abstract async connection class (#2734)

* make 'socket_timeout' and 'socket_connect_timeout' equivalent for TCP and UDS connections

* abstract asynio connection in analogy with the synchronous connection

---------

Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>

* Fix type hint for retry_on_error in async cluster (#2804)

* fix(asyncio.cluster): fixup retry_on_error type hint

This parameter accepts a list of _classes of Exceptions_, not a list of instantiated Exceptions. Fixup the type hint accordingly.

* chore: update changelog

---------

Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>

* Fix CI (#2809)

* Support JSON.MSET Command (#2766)

* support JSON.MERGE Command

* linters

* try with abc instead person

* change @skip_ifmodversion_lt to latest ReJSON 2.4.7

* change version

* fix test

* linters

* add async test

* Support JSON.MSET command

* trying to run CI

* linters

* add async test

* reminder do delete the integration changes

* delete the line from integration

* fix the interface

* change docstring

---------

Co-authored-by: Chayim <chayim@users.noreply.github.com>
Co-authored-by: dvora-h <dvora.heller@redis.com>

* Version 4.6.0 (#2810)

* master changes

* linters

* fix test_cwe_404 cluster test

---------

Co-authored-by: Thiago Bellini Ribeiro <hackedbellini@gmail.com>
Co-authored-by: woutdenolf <woutdenolf@users.sf.net>
Co-authored-by: Chayim <chayim@users.noreply.github.com>
Co-authored-by: shacharPash <93581407+shacharPash@users.noreply.github.com>
Co-authored-by: James R T <jamestiotio@gmail.com>
Co-authored-by: Mirek Długosz <miniopl+github@gmail.com>
Co-authored-by: Oran Avraham <252748+oranav@users.noreply.github.com>
Co-authored-by: mzdehbashi-github <85902780+mzdehbashi-github@users.noreply.github.com>
Co-authored-by: Tyler Hutcherson <tyler.hutcherson@redis.com>
Co-authored-by: Felipe Machado <462154+felipou@users.noreply.github.com>
Co-authored-by: AYMEN Mohammed <53928879+AYMENJD@users.noreply.github.com>
Co-authored-by: Marc Schöchlin <ms-github@256bit.org>
Co-authored-by: Marc Schöchlin <marc.schoechlin@flipapp.de>
Co-authored-by: Avasam <samuel.06@hotmail.com>
Co-authored-by: Markus Gerstel <2102431+Anthchirp@users.noreply.github.com>
Co-authored-by: Kristján Valur Jónsson <sweskman@gmail.com>
Co-authored-by: Nick Gerow <Nick.G.123@hotmail.com>
Co-authored-by: Nick Gerow <nick.gerow@enlightedinc.com>
Co-authored-by: Cristian Matache <cristianmatache@hotmail.com>
Co-authored-by: Anurag Bandyopadhyay <angbpy@gmail.com>
Co-authored-by: Anuragkillswitch <70265851+Anuragkillswitch@users.noreply.github.com>
Co-authored-by: Seongchuel Ahn <aciddust20@gmail.com>
Co-authored-by: Alibi <aliby.bbb@gmail.com>
Co-authored-by: Smit Parmar <smitraj333@gmail.com>
Co-authored-by: Brad MacPhee <macphee@gmail.com>
Co-authored-by: Igor Malinovskiy <u.glide@gmail.com>
Co-authored-by: Shahar Lev <shahar_lev@hotmail.com>
Co-authored-by: Vladimir Mihailenco <vladimir.webdev@gmail.com>
Co-authored-by: Kevin James <KevinJames@thekev.in>
  • Loading branch information
1 parent d453665 commit f2f8c34
Show file tree
Hide file tree
Showing 49 changed files with 2,341 additions and 1,156 deletions.
12 changes: 12 additions & 0 deletions CHANGES
Original file line number Diff line number Diff line change
@@ -1,3 +1,13 @@
* Fix incorrect redis.asyncio.Cluster type hint for `retry_on_error`
* Fix dead weakref in sentinel connection causing ReferenceError (#2767)
* Fix #2768, Fix KeyError: 'first-entry' in parse_xinfo_stream.
* Fix #2749, remove unnecessary __del__ logic to close connections.
* Fix #2754, adding a missing argument to SentinelManagedConnection
* Fix `xadd` command to accept non-negative `maxlen` including 0
* Revert #2104, #2673, add `disconnect_on_error` option to `read_response()` (issues #2506, #2624)
* Add `address_remap` parameter to `RedisCluster`
* Fix incorrect usage of once flag in async Sentinel
* asyncio: Fix memory leak caused by hiredis (#2693)
* Allow data to drain from async PythonParser when reading during a disconnect()
* Use asyncio.timeout() instead of async_timeout.timeout() for python >= 3.11 (#2602)
* Add test and fix async HiredisParser when reading during a disconnect() (#2349)
Expand Down Expand Up @@ -40,6 +50,8 @@
* Fix Sentinel.execute_command doesn't execute across the entire sentinel cluster bug (#2458)
* Added a replacement for the default cluster node in the event of failure (#2463)
* Fix for Unhandled exception related to self.host with unix socket (#2496)
* Improve error output for master discovery
* Make `ClusterCommandsProtocol` an actual Protocol

* 4.1.3 (Feb 8, 2022)
* Fix flushdb and flushall (#1926)
Expand Down
20 changes: 10 additions & 10 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,15 @@

## Introduction

First off, thank you for considering contributing to redis-py. We value
community contributions!
We appreciate your interest in considering contributing to redis-py.
Community contributions mean a lot to us.

## Contributions We Need
## Contributions we need

You may already know what you want to contribute \-- a fix for a bug you
You may already know how you'd like to contribute, whether it's a fix for a bug you
encountered, or a new feature your team wants to use.

If you don't know what to contribute, keep an open mind! Improving
If you don't know where to start, consider improving
documentation, bug triaging, and writing tutorials are all examples of
helpful contributions that mean less work for you.

Expand Down Expand Up @@ -166,19 +166,19 @@ When filing an issue, make sure to answer these five questions:
4. What did you expect to see?
5. What did you see instead?

## How to Suggest a Feature or Enhancement
## Suggest a feature or enhancement

If you'd like to contribute a new feature, make sure you check our
issue list to see if someone has already proposed it. Work may already
be under way on the feature you want -- or we may have rejected a
be underway on the feature you want or we may have rejected a
feature like it already.

If you don't see anything, open a new issue that describes the feature
you would like and how it should work.

## Code Review Process
## Code review process

The core team looks at Pull Requests on a regular basis. We will give
feedback as as soon as possible. After feedback, we expect a response
The core team regularly looks at pull requests. We will provide
feedback as as soon as possible. After receiving our feedback, please respond
within two weeks. After that time, we may close your PR if it isn't
showing any activity.
56 changes: 44 additions & 12 deletions docs/examples/connection_examples.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -267,28 +267,60 @@
}
],
"source": [
"from typing import Tuple, Union\n",
"from urllib.parse import ParseResult, urlencode, urlunparse\n",
"\n",
"import botocore.session\n",
"import redis\n",
"import boto3\n",
"import cachetools.func\n",
"from botocore.model import ServiceId\n",
"from botocore.signers import RequestSigner\n",
"from cachetools import TTLCache, cached\n",
"\n",
"class ElastiCacheIAMProvider(redis.CredentialProvider):\n",
" def __init__(self, user, endpoint, port=6379, region=\"us-east-1\"):\n",
" self.ec_client = boto3.client('elasticache')\n",
" def __init__(self, user, cluster_name, region=\"us-east-1\"):\n",
" self.user = user\n",
" self.endpoint = endpoint\n",
" self.port = port\n",
" self.cluster_name = cluster_name\n",
" self.region = region\n",
"\n",
" session = botocore.session.get_session()\n",
" self.request_signer = RequestSigner(\n",
" ServiceId(\"elasticache\"),\n",
" self.region,\n",
" \"elasticache\",\n",
" \"v4\",\n",
" session.get_credentials(),\n",
" session.get_component(\"event_emitter\"),\n",
" )\n",
"\n",
" # Generated IAM tokens are valid for 15 minutes\n",
" @cached(cache=TTLCache(maxsize=128, ttl=900))\n",
" def get_credentials(self) -> Union[Tuple[str], Tuple[str, str]]:\n",
" @cachetools.func.ttl_cache(maxsize=128, ttl=15 * 60) # 15m\n",
" def get_iam_auth_token(user, endpoint, port, region):\n",
" return self.ec_client.generate_iam_auth_token(user, endpoint, port, region)\n",
" iam_auth_token = get_iam_auth_token(self.endpoint, self.port, self.user, self.region)\n",
" return iam_auth_token\n",
" query_params = {\"Action\": \"connect\", \"User\": self.user}\n",
" url = urlunparse(\n",
" ParseResult(\n",
" scheme=\"https\",\n",
" netloc=self.cluster_name,\n",
" path=\"/\",\n",
" query=urlencode(query_params),\n",
" params=\"\",\n",
" fragment=\"\",\n",
" )\n",
" )\n",
" signed_url = self.request_signer.generate_presigned_url(\n",
" {\"method\": \"GET\", \"url\": url, \"body\": {}, \"headers\": {}, \"context\": {}},\n",
" operation_name=\"connect\",\n",
" expires_in=900,\n",
" region_name=self.region,\n",
" )\n",
" # RequestSigner only seems to work if the URL has a protocol, but\n",
" # Elasticache only accepts the URL without a protocol\n",
" # So strip it off the signed URL before returning\n",
" return (self.user, signed_url.removeprefix(\"https://\"))\n",
"\n",
"username = \"barshaul\"\n",
"cluster_name = \"test-001\"\n",
"endpoint = \"test-001.use1.cache.amazonaws.com\"\n",
"creds_provider = ElastiCacheIAMProvider(user=username, endpoint=endpoint)\n",
"creds_provider = ElastiCacheIAMProvider(user=username, cluster_name=cluster_name)\n",
"user_connection = redis.Redis(host=endpoint, port=6379, credential_provider=creds_provider)\n",
"user_connection.ping()"
]
Expand Down
3 changes: 1 addition & 2 deletions docs/examples/opentelemetry/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,11 @@

import time

import redis
import uptrace
from opentelemetry import trace
from opentelemetry.instrumentation.redis import RedisInstrumentor

import redis

tracer = trace.get_tracer("app_or_package_name", "1.0.0")


Expand Down
Loading

0 comments on commit f2f8c34

Please sign in to comment.