Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix issue 2349: Let async HiredisParser finish parsing after a Connection.disconnect() #2557

Merged
merged 6 commits into from
Feb 6, 2023

Conversation

kristjanvalur
Copy link
Contributor

@kristjanvalur kristjanvalur commented Jan 20, 2023

Pull Request check-list

Please make sure to review and check all of these items:

  • Does $ tox pass with this change (including linting)?
  • Do the CI tests pass with this change (enable it first in your forked repo and wait for the github action build to finish)?
  • Is the new or changed code fully tested?
  • Is a documentation update included (if this change modifies existing APIs, or introduces new ones)?
  • Is there an example added to the examples folder (if applicable)?
  • Was the change added to CHANGES file?

NOTE: these things are not required to open a PR and can be done
afterwards / while the PR is open.

Description of change

This change addresses an issue raised in issue #2349.

It ensures that an async Connection.read_response() call can succeed even if a simultaneous Connection.disconnect()
call is issued. Socket disconnects are graceful in that one closes the send direction and can then wait for any data
in transit to arrive, until an EOF is delivered from the remote side. In asyncio parlance this means that to gracefully close a socket, you close the send stream and can then read from the receive stream until you reach an End Of File.

This change makes sure that the internal BaseParser state is maintained so that it can continue to request and parse chunks for as long as it can. Previously, the HiredisParser would get its internal state cleared immediately, and this would cause the read, in progress, to fail with an unexpected error. Now an ongoing read_response() will continue until it succeeds or encounters an error such as EOF.

A new read_response() issued after a disconnect() call has been issued (but before it completes) will immediately fail, howerver. This is to maintain previous semantics, even though we could simply allow it to go through and fail later when it attemts to read from a closed connection.

@kristjanvalur kristjanvalur force-pushed the kristjan/issue_2349 branch 7 times, most recently from 3752994 to 78c81f4 Compare January 20, 2023 15:02
@kristjanvalur kristjanvalur changed the title Kristjan/issue 2349 Fix issue 2349: Let async HiredisParser finish parsing after a Connection.disconnect() Jan 20, 2023
@kristjanvalur kristjanvalur marked this pull request as ready for review January 20, 2023 15:16
@codecov-commenter
Copy link

codecov-commenter commented Jan 20, 2023

Codecov Report

Base: 92.25% // Head: 92.28% // Increases project coverage by +0.02% 🎉

Coverage data is based on head (4d45a5e) compared to base (9e00b91).
Patch coverage: 100.00% of modified lines in pull request are covered.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2557      +/-   ##
==========================================
+ Coverage   92.25%   92.28%   +0.02%     
==========================================
  Files         115      115              
  Lines       29692    29734      +42     
==========================================
+ Hits        27393    27440      +47     
+ Misses       2299     2294       -5     
Impacted Files Coverage Δ
redis/asyncio/connection.py 87.62% <100.00%> (+0.12%) ⬆️
tests/test_asyncio/test_connection.py 98.08% <100.00%> (+0.69%) ⬆️
tests/test_cluster.py 96.86% <0.00%> (+0.10%) ⬆️
tests/test_asyncio/test_search.py 98.63% <0.00%> (+0.34%) ⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@kristjanvalur
Copy link
Contributor Author

random test failure is random.

@dvora-h dvora-h added the bug Bug label Jan 22, 2023
@dvora-h
Copy link
Collaborator

dvora-h commented Feb 6, 2023

@kristjanvalur Thanks for this PR! Can you please resolve the conflicts so I can merge it?

@dvora-h dvora-h merged commit e7306aa into redis:master Feb 6, 2023
@kristjanvalur kristjanvalur deleted the kristjan/issue_2349 branch February 7, 2023 08:37
@chayim chayim mentioned this pull request Jul 31, 2023
6 tasks
chayim pushed a commit that referenced this pull request Aug 31, 2023
Co-authored-by: Viktor Ivanov <viktor@infogrid.io>
Co-authored-by: Sergey Prokazov <sergey.prokazov@redis.com>
Co-authored-by: Anuragkillswitch <70265851+Anuragkillswitch@users.noreply.github.com>
Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>
Co-authored-by: Alex Schmitz <aschmitz@box.com>
Co-authored-by: Alex Schmitz <alex.schmitz@gmail.com>
Co-authored-by: Chayim <chayim@users.noreply.github.com>
Co-authored-by: Bar Shaul <88437685+barshaul@users.noreply.github.com>
Co-authored-by: CrimsonGlory <CrimsonGlory@users.noreply.github.com>
Co-authored-by: Raymond Yin <raymond@tryevergreen.com>
Co-authored-by: zach.lee <zach.lee@sendbird.com>
Co-authored-by: James R T <jamestiotio@gmail.com>
Co-authored-by: dvora-h <dvora.heller@redis.com>
Co-authored-by: Marc Schöchlin <marc.schoechlin@flipapp.de>
Co-authored-by: Nick Gerow <nick.gerow@enlightedinc.com>
Co-authored-by: Igor Malinovskiy <u.glide@gmail.com>
Co-authored-by: Chayim I. Kirshen <c@kirshen.com>
Co-authored-by: Leibale Eidelman <me@leibale.com>
Co-authored-by: Thiago Bellini Ribeiro <hackedbellini@gmail.com>
Co-authored-by: woutdenolf <woutdenolf@users.sf.net>
Co-authored-by: shacharPash <93581407+shacharPash@users.noreply.github.com>
Co-authored-by: Mirek Długosz <miniopl+github@gmail.com>
Co-authored-by: Oran Avraham <252748+oranav@users.noreply.github.com>
Co-authored-by: mzdehbashi-github <85902780+mzdehbashi-github@users.noreply.github.com>
Co-authored-by: Tyler Hutcherson <tyler.hutcherson@redis.com>
Co-authored-by: Felipe Machado <462154+felipou@users.noreply.github.com>
Co-authored-by: AYMEN Mohammed <53928879+AYMENJD@users.noreply.github.com>
Co-authored-by: Marc Schöchlin <ms-github@256bit.org>
Co-authored-by: Avasam <samuel.06@hotmail.com>
Co-authored-by: Markus Gerstel <2102431+Anthchirp@users.noreply.github.com>
Co-authored-by: Kristján Valur Jónsson <sweskman@gmail.com>
Co-authored-by: Nick Gerow <Nick.G.123@hotmail.com>
Co-authored-by: Cristian Matache <cristianmatache@hotmail.com>
Co-authored-by: Anurag Bandyopadhyay <angbpy@gmail.com>
Co-authored-by: Seongchuel Ahn <aciddust20@gmail.com>
Co-authored-by: Alibi <aliby.bbb@gmail.com>
Co-authored-by: Smit Parmar <smitraj333@gmail.com>
Co-authored-by: Brad MacPhee <macphee@gmail.com>
Co-authored-by: Shahar Lev <shahar_lev@hotmail.com>
Co-authored-by: Vladimir Mihailenco <vladimir.webdev@gmail.com>
Co-authored-by: Kevin James <KevinJames@thekev.in>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: David Pacsuta <34983281+ant1fact@users.noreply.github.com>
Co-authored-by: Rich Bowen <rbowen@rcbowen.com>
Co-authored-by: gmbnomis <gmbnomis@users.noreply.github.com>
Co-authored-by: Vivanov98 <66319645+Vivanov98@users.noreply.github.com>
Co-authored-by: Kosuke <kosuke.zhang@gmail.com>
Co-authored-by: Sergey Prokazov <prokazov@users.noreply.github.com>
Co-authored-by: jmcbailey <jmcbailey@gmail.com>
Co-authored-by: Galtozzy <14139502+Galtozzy@users.noreply.github.com>
Co-authored-by: Abhishek Kumar Sinha <aksinha334@gmail.com>
Co-authored-by: Eom Taegyung "Iggy <iggy.eom@sendbird.com>
Co-authored-by: Mehdi ABAAKOUK <sileht@sileht.net>
Co-authored-by: Dongkeun Lee <3315213+zakaf@users.noreply.github.com>
Co-authored-by: woutdenolf <wout.de_nolf@esrf.eu>
Co-authored-by: Kurt McKee <contactme@kurtmckee.org>
Co-authored-by: Juraj Páll <palljuraj1@gmail.com>
Co-authored-by: Joan Fontanals <jfontanalsmartinez@gmail.com>
Co-authored-by: Stanislav Zmiev <zmievsa@gmail.com>
fix (#2566)
Fix unlink in cluster pipeline (#2562)
Fix issue 2540: Synchronise concurrent command calls to single-client mode. (#2568)
Fix: tuple function cannot be passed more than one argument (#2573)
Fix issue 2567: NoneType check before raising exception (#2569)
Fix issue 2349: Let async HiredisParser finish parsing after a Connection.disconnect() (#2557)
Fix issue with `pack_commands` returning an empty byte sequence (#2416)
Fix #2581 UnixDomainSocketConnection' object has no attribute '_command_packer' (#2583)
Fix #2581 UnixDomainSocketConnection' object has no attribute '_command_packer' .
Fix for `lpop` and `rpop` return typing (#2590)
Fixed CredentialsProvider examples (#2587)
Fixed issue #2598 - make Document class subscriptable
fix: replace async_timeout by asyncio.timeout (#2602)
Fix behaviour of async PythonParser to match RedisParser as for issue #2349 (#2582)
Fix (#2641)
fix: do not use asyncio's timeout lib before 3.11.2 (#2659)
Fix issue 2660: PytestUnraisableExceptionWarning from asycio client (#2669)
Fixing cancelled async futures (#2666)
Fix async (#2673)
Fix memory leak caused by hiredis (#2693) (#2694)
Fix incorrect usage of once flag in async Sentinel (#2718)
Fix topk list example. (#2724)
Fix `ClusterCommandProtocol` not itself being marked as a protocol (#2729)
Fix potential race condition during disconnection (#2719)
fix CI (#2748)
fix parse_slowlog_get (#2732)
fixes for issue #1128
fix create single_connection_client from url (#2752)
Fix `xadd` allow non negative maxlen (#2739)
Fix JSON.MERGE Summary (#2786)
Fixed key error in parse_xinfo_stream (#2788)
Fix dead weakref in sentinel connection causing ReferenceError (#2767) (#2771)
Fix dead weakref in sentinel conn (#2767)
fix redirects and some small cleanups (#2801)
Fix type hint for retry_on_error in async cluster (#2804)
Fix CI (#2809)
Fix async client with resp3 (#2657)
Fix `COMMAND` response in resp3 (redis 7+) (#2740)
Fix protocol version checking (#2737)
Fix parse resp3 dict response: don't use dict comprehension (#2757)
Fixing asyncio import (#2759)
fix (#2799)
fix async tests (#2806)
Fix socket garbage collection (#2859)
Fixing doc builds (#2869)
Fix a duplicate word in `CONTRIBUTING.md` (#2848)
Fix timeout retrying on Redis pipeline execution (#2812)
Fix type hints in SearchCommands (#2817)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Bug
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants