ChunkedEncodingError on mocked HEAD requests with urllib3 >= 2.0.0

[progval]

This code:

import functools

import requests
import requests_mock


def f(request, context):
    context.headers["Content-Length"] = "300810"
    return None

with requests_mock.Mocker() as m:
    m.head(
        url="https://hackage.haskell.org/package/aeson-2.1.0.0/aeson-2.1.0.0.tar.gz",
        status_code=200,
        text=f,
    )

    requests.head("https://hackage.haskell.org/package/aeson-2.1.0.0/aeson-2.1.0.0.tar.gz")

stopped working since urllib3 released version 2.0.2 two days ago (2.0.0 and 2.0.1 were yanked).

Here is how to reproduce:

$ python3 -m venv venv
$ . ./venv/bin/activate
(venv) dev@desktop5:/tmp$ pip3 install requests requests-mock
Collecting requests
  Using cached requests-2.30.0-py3-none-any.whl (62 kB)
Collecting requests-mock
  Using cached requests_mock-1.10.0-py2.py3-none-any.whl (28 kB)
Collecting urllib3<3,>=1.21.1
  Using cached urllib3-2.0.2-py3-none-any.whl (123 kB)
Collecting charset-normalizer<4,>=2
  Using cached charset_normalizer-3.1.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (199 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2022.12.7-py3-none-any.whl (155 kB)
Collecting idna<4,>=2.5
  Using cached idna-3.4-py3-none-any.whl (61 kB)
Collecting six
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: urllib3, charset-normalizer, certifi, idna, requests, six, requests-mock
Successfully installed certifi-2022.12.7 charset-normalizer-3.1.0 idna-3.4 requests-2.30.0 requests-mock-1.10.0 six-1.16.0 urllib3-2.0.2
$ python3 reqmock_repro.py 
Traceback (most recent call last):
  File "/tmp/venv/lib/python3.9/site-packages/urllib3/response.py", line 705, in _error_catcher
    yield
  File "/tmp/venv/lib/python3.9/site-packages/urllib3/response.py", line 830, in _raw_read
    raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
urllib3.exceptions.IncompleteRead: IncompleteRead(0 bytes read, 300810 more expected)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/tmp/venv/lib/python3.9/site-packages/requests/models.py", line 816, in generate
    yield from self.raw.stream(chunk_size, decode_content=True)
  File "/tmp/venv/lib/python3.9/site-packages/urllib3/response.py", line 935, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/tmp/venv/lib/python3.9/site-packages/urllib3/response.py", line 874, in read
    data = self._raw_read(amt)
  File "/tmp/venv/lib/python3.9/site-packages/urllib3/response.py", line 830, in _raw_read
    raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
  File "/usr/lib/python3.9/contextlib.py", line 135, in __exit__
    self.gen.throw(type, value, traceback)
  File "/tmp/venv/lib/python3.9/site-packages/urllib3/response.py", line 722, in _error_catcher
    raise ProtocolError(f"Connection broken: {e!r}", e) from e
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(0 bytes read, 300810 more expected)', IncompleteRead(0 bytes read, 300810 more expected))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/reqmock_repro.py", line 18, in <module>
    requests.head("https://hackage.haskell.org/package/aeson-2.1.0.0/aeson-2.1.0.0.tar.gz")
  File "/tmp/venv/lib/python3.9/site-packages/requests/api.py", line 100, in head
    return request("head", url, **kwargs)
  File "/tmp/venv/lib/python3.9/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/tmp/venv/lib/python3.9/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/tmp/venv/lib/python3.9/site-packages/requests_mock/mocker.py", line 185, in _fake_send
    return _original_send(session, request, **kwargs)
  File "/tmp/venv/lib/python3.9/site-packages/requests/sessions.py", line 745, in send
    r.content
  File "/tmp/venv/lib/python3.9/site-packages/requests/models.py", line 899, in content
    self._content = b"".join(self.iter_content(CONTENT_CHUNK_SIZE)) or b""
  File "/tmp/venv/lib/python3.9/site-packages/requests/models.py", line 818, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read, 300810 more expected)', IncompleteRead(0 bytes read, 300810 more expected))

but it works if we require an older version of urllib3:

(venv) $ pip3 install 'urllib3<2.0.0'
Collecting urllib3<2.0.0
  Using cached urllib3-1.26.15-py2.py3-none-any.whl (140 kB)
Installing collected packages: urllib3
  Attempting uninstall: urllib3
    Found existing installation: urllib3 2.0.2
    Uninstalling urllib3-2.0.2:
      Successfully uninstalled urllib3-2.0.2
Successfully installed urllib3-1.26.15
(venv) $ python3 reqmock_repro.py 
(venv) $

[wardal]

Same with .delete() method, when used json argument. Without using json or in topic starter example without using text argument, all works fine. When adding some data to request - ChunkedEncodingError raised.

[mgorny]

This is especially cumbersome since requests-mock's test suite seems to pass with urllib3-2 just fine, and you end up guessing why random packages suddenly started failing (e.g. a lot of OpenStack).

[jamielennox]

released as 1.11.0 - thanks for the info and sorry it took me some time to fix it.

[progval]

thanks!

[mgorny]

Thanks! I have tested the new version against urllib3-2 and a few packages whose test suites have failed previously, and I think the problem's solved. I see some test failures from requests-cache, keystoneauth1 and python-glanceclient but they don't seem to be related to requests-mock.