Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ClientWebSocketResponse.close_code will be None when there are concurrent async task receiving data and closing connection #7306

Closed
wenleix opened this issue May 31, 2023 · 1 comment · Fixed by #7680
Labels

Comments

@wenleix
Copy link

wenleix commented May 31, 2023

Describe the bug

If one asyncio task is waiting on receiving data and another asyncio task is closing the connection. The ClientWebSocketResponse.close_code will be None after connection closed.

It looks like the following is happening:

  1. close() called and feed WS_CLOSING_MESSAGE:
    if self._waiting is not None and not self._closed:
    self._reader.feed_data(WS_CLOSING_MESSAGE, 0)
    await self._waiting
  2. receive() received WS_CLOSING_MESSAGE , mark self._closing to be True:
    elif msg.type == WSMsgType.CLOSING:
    self._closing = True
  3. close() (after sent out the close message to server) find self._closing is True, close the response and return, WITHOUT setting self._close_code:
    if self._closing:
    self._response.close()
    return True

To Reproduce

See the steps in https://github.com/wenleix/aiohttp_recv_upon_close

Expected behavior

Expect to have closed code 1000, i.e.

Waiting for message
Closed code: 1000

Logs/tracebacks

The actual result is :

Waiting for message
Closed code: None


### Python Version

```console
$ python --version

Python 3.9.13


### aiohttp Version

```console
$ python -m pip show aiohttp

Name: aiohttp
Version: 3.8.4
Summary: Async http client/server framework (asyncio)
Home-page: https://github.com/aio-libs/aiohttp
Author:
Author-email:
License: Apache 2
Requires: aiosignal, async-timeout, attrs, charset-normalizer, frozenlist, multidict, yarl
Required-by:


### multidict Version

```console
$ python -m pip show multidict

Name: multidict
Version: 6.0.4
Summary: multidict implementation
Home-page: https://github.com/aio-libs/multidict
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: Apache 2
Requires:
Required-by: aiohttp, yarl


### yarl Version

```console
$ python -m pip show yarl

Name: yarl
Version: 1.9.2
Summary: Yet another URL library
Home-page: https://github.com/aio-libs/yarl/
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: Apache-2.0
Requires: idna, multidict
Required-by: aiohttp


### OS

macOS

### Related component

Client

### Additional context

_No response_

### Code of Conduct

- [X] I agree to follow the aio-libs Code of Conduct
@wenleix wenleix added the bug label May 31, 2023
@wenleix wenleix changed the title ClientWebSocketResponse.close_code will be None when there are concurrent async task receiving data and connection close ClientWebSocketResponse.close_code will be None when there are concurrent async task receiving data and closing connection May 31, 2023
@Dreamsorcerer
Copy link
Member

Looks to me like this should be changed:

- if self._waiting is not None and not self._closed:
+ if self._waiting is not None and not self._closing:
+     self._closing = True

Otherwise, multiple close() calls could send multiple CLOSING messages.

Then:

- if self._closing:
+ if self._close_code:

So, in error cases, where the code is already set, it closes immediately. In normal closing, it would continue to the end of the closing logic.

Dreamsorcerer pushed a commit that referenced this issue Oct 9, 2023
ttsia added a commit to ttsia/aiohttp that referenced this issue Oct 10, 2023
…in concurrent closing scenario (aio-libs#7680)

(cherry picked from commit 30850ba)
Dreamsorcerer pushed a commit that referenced this issue Oct 11, 2023
…nse.close_code correctly in concu… (#7690)

This is a backport of PR #7680
as merged into master
(30850ba).

Fixes the issue when If one asyncio task is waiting on receiving data
and another asyncio task is closing the connection. The
ClientWebSocketResponse.close_code will be None after connection closed.
renovate bot referenced this issue in allenporter/pyrainbird Nov 20, 2023
[![Mend Renovate logo
banner](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
| [aiohttp](https://github.com/aio-libs/aiohttp) | `==3.8.6` ->
`==3.9.0` |
[![age](https://developer.mend.io/api/mc/badges/age/pypi/aiohttp/3.9.0?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![adoption](https://developer.mend.io/api/mc/badges/adoption/pypi/aiohttp/3.9.0?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![passing](https://developer.mend.io/api/mc/badges/compatibility/pypi/aiohttp/3.8.6/3.9.0?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![confidence](https://developer.mend.io/api/mc/badges/confidence/pypi/aiohttp/3.8.6/3.9.0?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|

---

### Release Notes

<details>
<summary>aio-libs/aiohttp (aiohttp)</summary>

###
[`v3.9.0`](https://github.com/aio-libs/aiohttp/blob/HEAD/CHANGES.rst#390-2023-11-18)

[Compare
Source](https://github.com/aio-libs/aiohttp/compare/v3.8.6...v3.9.0)

\==================

## Features

- Introduced `AppKey` for static typing support of `Application`
storage.
See
https://docs.aiohttp.org/en/stable/web_advanced.html#application-s-config

    `#&#8203;5864 <https://github.com/aio-libs/aiohttp/issues/5864>`\_

- Added a graceful shutdown period which allows pending tasks to
complete before the application's cleanup is called.
The period can be adjusted with the `shutdown_timeout` parameter. -- by
:user:`Dreamsorcerer`.
See
https://docs.aiohttp.org/en/latest/web_advanced.html#graceful-shutdown

    `#&#8203;7188 <https://github.com/aio-libs/aiohttp/issues/7188>`\_

- Added `handler_cancellation
<https://docs.aiohttp.org/en/stable/web_advanced.html#web-handler-cancellation>`\_
parameter to cancel web handler on client disconnection. -- by
:user:`mosquito`
This (optionally) reintroduces a feature removed in a previous release.
Recommended for those looking for an extra level of protection against
denial-of-service attacks.

    `#&#8203;7056 <https://github.com/aio-libs/aiohttp/issues/7056>`\_

- Added support for setting response header parameters `max_line_size`
and `max_field_size`.

    `#&#8203;2304 <https://github.com/aio-libs/aiohttp/issues/2304>`\_

- Added `auto_decompress` parameter to `ClientSession.request` to
override `ClientSession._auto_decompress`. -- by :user:`Daste745`

    `#&#8203;3751 <https://github.com/aio-libs/aiohttp/issues/3751>`\_

-   Changed `raise_for_status` to allow a coroutine.

    `#&#8203;3892 <https://github.com/aio-libs/aiohttp/issues/3892>`\_

- Added client brotli compression support (optional with runtime check).

    `#&#8203;5219 <https://github.com/aio-libs/aiohttp/issues/5219>`\_

- Added `client_max_size` to `BaseRequest.clone()` to allow overriding
the request body size. -- :user:`anesabml`.

    `#&#8203;5704 <https://github.com/aio-libs/aiohttp/issues/5704>`\_

-   Added a middleware type alias `aiohttp.typedefs.Middleware`.

    `#&#8203;5898 <https://github.com/aio-libs/aiohttp/issues/5898>`\_

- Exported `HTTPMove` which can be used to catch any redirection request
    that has a location -- :user:`dreamsorcerer`.

    `#&#8203;6594 <https://github.com/aio-libs/aiohttp/issues/6594>`\_

- Changed the `path` parameter in `web.run_app()` to accept a
`pathlib.Path` object.

    `#&#8203;6839 <https://github.com/aio-libs/aiohttp/issues/6839>`\_

- Performance: Skipped filtering `CookieJar` when the jar is empty or
all cookies have expired.

    `#&#8203;7819 <https://github.com/aio-libs/aiohttp/issues/7819>`\_

- Performance: Only check origin if insecure scheme and there are
origins to treat as secure, in `CookieJar.filter_cookies()`.

    `#&#8203;7821 <https://github.com/aio-libs/aiohttp/issues/7821>`\_

- Performance: Used timestamp instead of `datetime` to achieve faster
cookie expiration in `CookieJar`.

    `#&#8203;7824 <https://github.com/aio-libs/aiohttp/issues/7824>`\_

- Added support for passing a custom server name parameter to HTTPS
connection.

    `#&#8203;7114 <https://github.com/aio-libs/aiohttp/issues/7114>`\_

- Added support for using Basic Auth credentials from :file:`.netrc`
file when making HTTP requests with the
:py:class:`~aiohttp.ClientSession` `trust_env` argument is set to
`True`. -- by :user:`yuvipanda`.

    `#&#8203;7131 <https://github.com/aio-libs/aiohttp/issues/7131>`\_

-   Turned access log into no-op when the logger is disabled.

    `#&#8203;7240 <https://github.com/aio-libs/aiohttp/issues/7240>`\_

- Added typing information to `RawResponseMessage`. -- by
:user:`Gobot1234`

    `#&#8203;7365 <https://github.com/aio-libs/aiohttp/issues/7365>`\_

- Removed `async-timeout` for Python 3.11+ (replaced with
`asyncio.timeout()` on newer releases).

    `#&#8203;7502 <https://github.com/aio-libs/aiohttp/issues/7502>`\_

- Added support for `brotlicffi` as an alternative to `brotli` (fixing
Brotli support on PyPy).

    `#&#8203;7611 <https://github.com/aio-libs/aiohttp/issues/7611>`\_

- Added `WebSocketResponse.get_extra_info()` to access a protocol
transport's extra info.

    `#&#8203;7078 <https://github.com/aio-libs/aiohttp/issues/7078>`\_

-   Allow `link` argument to be set to None/empty in HTTP 451 exception.

    `#&#8203;7689 <https://github.com/aio-libs/aiohttp/issues/7689>`\_

## Bugfixes

- Implemented stripping the trailing dots from fully-qualified domain
names in `Host` headers and TLS context when acting as an HTTP client.
This allows the client to connect to URLs with FQDN host name like
`https://example.com./`.
    \-- by :user:`martin-sucha`.

    `#&#8203;3636 <https://github.com/aio-libs/aiohttp/issues/3636>`\_

- Fixed client timeout not working when incoming data is always
available without waiting. -- by :user:`Dreamsorcerer`.

    `#&#8203;5854 <https://github.com/aio-libs/aiohttp/issues/5854>`\_

- Fixed `readuntil` to work with a delimiter of more than one character.

    `#&#8203;6701 <https://github.com/aio-libs/aiohttp/issues/6701>`\_

-   Added `__repr__` to `EmptyStreamReader` to avoid `AttributeError`.

    `#&#8203;6916 <https://github.com/aio-libs/aiohttp/issues/6916>`\_

-   Fixed bug when using `TCPConnector` with `ttl_dns_cache=0`.

    `#&#8203;7014 <https://github.com/aio-libs/aiohttp/issues/7014>`\_

- Fixed response returned from expect handler being thrown away. -- by
:user:`Dreamsorcerer`

    `#&#8203;7025 <https://github.com/aio-libs/aiohttp/issues/7025>`\_

- Avoided raising `UnicodeDecodeError` in multipart and in HTTP headers
parsing.

    `#&#8203;7044 <https://github.com/aio-libs/aiohttp/issues/7044>`\_

- Changed `sock_read` timeout to start after writing has finished,
avoiding read timeouts caused by an unfinished write. -- by
:user:`dtrifiro`

    `#&#8203;7149 <https://github.com/aio-libs/aiohttp/issues/7149>`\_

-   Fixed missing query in tracing method URLs when using `yarl` 1.9+.

    `#&#8203;7259 <https://github.com/aio-libs/aiohttp/issues/7259>`\_

- Changed max 32-bit timestamp to an aware datetime object, for
consistency with the non-32-bit one, and to avoid a `DeprecationWarning`
on Python 3.12.

    `#&#8203;7302 <https://github.com/aio-libs/aiohttp/issues/7302>`\_

- Fixed `EmptyStreamReader.iter_chunks()` never ending. -- by
:user:`mind1m`

    `#&#8203;7616 <https://github.com/aio-libs/aiohttp/issues/7616>`\_

- Fixed a rare `RuntimeError: await wasn't used with future` exception.
-- by :user:`stalkerg`

    `#&#8203;7785 <https://github.com/aio-libs/aiohttp/issues/7785>`\_

-   Fixed issue with insufficient HTTP method and version validation.

    `#&#8203;7700 <https://github.com/aio-libs/aiohttp/issues/7700>`\_

-   Added check to validate that absolute URIs have schemes.

    `#&#8203;7712 <https://github.com/aio-libs/aiohttp/issues/7712>`\_

- Fixed unhandled exception when Python HTTP parser encounters unpaired
Unicode surrogates.

    `#&#8203;7715 <https://github.com/aio-libs/aiohttp/issues/7715>`\_

- Updated parser to disallow invalid characters in header field names
and stop accepting LF as a request line separator.

    `#&#8203;7719 <https://github.com/aio-libs/aiohttp/issues/7719>`\_

-   Fixed Python HTTP parser not treating 204/304/1xx as an empty body.

    `#&#8203;7755 <https://github.com/aio-libs/aiohttp/issues/7755>`\_

-   Ensure empty body response for 1xx/204/304 per RFC 9112 sec 6.3.

    `#&#8203;7756 <https://github.com/aio-libs/aiohttp/issues/7756>`\_

- Fixed an issue when a client request is closed before completing a
chunked payload. -- by :user:`Dreamsorcerer`

    `#&#8203;7764 <https://github.com/aio-libs/aiohttp/issues/7764>`\_

-   Edge Case Handling for ResponseParser for missing reason value.

    `#&#8203;7776 <https://github.com/aio-libs/aiohttp/issues/7776>`\_

- Fixed `ClientWebSocketResponse.close_code` being erroneously set to
`None` when there are concurrent async tasks receiving data and closing
the connection.

    `#&#8203;7306 <https://github.com/aio-libs/aiohttp/issues/7306>`\_

-   Added HTTP method validation.

    `#&#8203;6533 <https://github.com/aio-libs/aiohttp/issues/6533>`\_

- Fixed arbitrary sequence types being allowed to inject values via
version parameter. -- by :user:`Dreamsorcerer`

    `#&#8203;7835 <https://github.com/aio-libs/aiohttp/issues/7835>`\_

- Performance: Fixed increase in latency with small messages from
websocket compression changes.

    `#&#8203;7797 <https://github.com/aio-libs/aiohttp/issues/7797>`\_

## Improved Documentation

- Fixed the `ClientResponse.release`'s type in the doc. Changed from
`comethod` to `method`.

    `#&#8203;5836 <https://github.com/aio-libs/aiohttp/issues/5836>`\_

- Added information on behavior of base_url parameter in
`ClientSession`.

    `#&#8203;6647 <https://github.com/aio-libs/aiohttp/issues/6647>`\_

-   Fixed `ClientResponseError` docs.

    `#&#8203;6700 <https://github.com/aio-libs/aiohttp/issues/6700>`\_

-   Updated Redis code examples to follow the latest API.

    `#&#8203;6907 <https://github.com/aio-libs/aiohttp/issues/6907>`\_

- Added a note about possibly needing to update headers when using
`on_response_prepare`. -- by :user:`Dreamsorcerer`

    `#&#8203;7283 <https://github.com/aio-libs/aiohttp/issues/7283>`\_

- Completed `trust_env` parameter description to honor `wss_proxy`,
`ws_proxy` or `no_proxy` env.

    `#&#8203;7325 <https://github.com/aio-libs/aiohttp/issues/7325>`\_

- Expanded SSL documentation with more examples (e.g. how to use
certifi). -- by :user:`Dreamsorcerer`

    `#&#8203;7334 <https://github.com/aio-libs/aiohttp/issues/7334>`\_

-   Fix, update, and improve client exceptions documentation.

    `#&#8203;7733 <https://github.com/aio-libs/aiohttp/issues/7733>`\_

## Deprecations and Removals

-   Added `shutdown_timeout` parameter to `BaseRunner`, while
deprecating `shutdown_timeout` parameter from `BaseSite`. -- by
:user:`Dreamsorcerer`

    `#&#8203;7718 <https://github.com/aio-libs/aiohttp/issues/7718>`\_

-   Dropped Python 3.6 support.

    `#&#8203;6378 <https://github.com/aio-libs/aiohttp/issues/6378>`\_

-   Dropped Python 3.7 support. -- by :user:`Dreamsorcerer`

    `#&#8203;7336 <https://github.com/aio-libs/aiohttp/issues/7336>`\_

- Removed support for abandoned `tokio` event loop. -- by
:user:`Dreamsorcerer`

    `#&#8203;7281 <https://github.com/aio-libs/aiohttp/issues/7281>`\_

## Misc

-   Made `print` argument in `run_app()` optional.

    `#&#8203;3690 <https://github.com/aio-libs/aiohttp/issues/3690>`\_

-   Improved performance of `ceil_timeout` in some cases.

    `#&#8203;6316 <https://github.com/aio-libs/aiohttp/issues/6316>`\_

- Changed importing Gunicorn to happen on-demand, decreasing import time
by ~53%. -- :user:`Dreamsorcerer`

    `#&#8203;6591 <https://github.com/aio-libs/aiohttp/issues/6591>`\_

- Improved import time by replacing `http.server` with
`http.HTTPStatus`.

    `#&#8203;6903 <https://github.com/aio-libs/aiohttp/issues/6903>`\_

- Fixed annotation of `ssl` parameter to disallow `True`. -- by
:user:`Dreamsorcerer`.

    `#&#8203;7335 <https://github.com/aio-libs/aiohttp/issues/7335>`\_

***

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined),
Automerge - At any time (no schedule defined).

🚦 **Automerge**: Enabled.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR has been generated by [Mend
Renovate](https://www.mend.io/free-developer-tools/renovate/). View
repository job log
[here](https://developer.mend.io/github/allenporter/pyrainbird).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy41OS44IiwidXBkYXRlZEluVmVyIjoiMzcuNTkuOCIsInRhcmdldEJyYW5jaCI6Im1haW4ifQ==-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
xiangxli pushed a commit to xiangxli/aiohttp that referenced this issue Dec 4, 2023
…ntWebSocketResponse.close_code correctly in concu… (aio-libs#7690)

This is a backport of PR aio-libs#7680
as merged into master
(aio-libs@30850ba).

Fixes the issue when If one asyncio task is waiting on receiving data
and another asyncio task is closing the connection. The
ClientWebSocketResponse.close_code will be None after connection closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
2 participants