Updating dependency requests to 2.32.2 will fail with Not supported URL scheme http+unix

[henkjan-van-wijk]

Context

• OS and version used: Ubuntu 22.04
• Python version: 3.12.2
• pip version: 24.0
• list of installed packages:
  Package Version

---

annotated-types 0.7.0
azure-iot-device 2.13.0
certifi 2024.2.2
charset-normalizer 3.3.2
deprecation 2.1.0
idna 3.7
janus 1.0.0
packaging 24.0
paho-mqtt 1.6.1
pip 24.0
pydantic 2.7.1
pydantic_core 2.18.2
PySocks 1.7.1
requests 2.32.2
requests-unixsocket 0.3.0
typing_extensions 4.11.0
urllib3 1.26.18

Description of the issue

Today we updated the dependencies on a test iot edge module and saw that the module no longer could connect to the edgeHub using IoTHubModuleClient.create_from_edge_environment().

After some testing to check which changes were made, it seems to boil down to the requests package which has been updated May 20, 2024 to 2.32.0 with some subsequent updates to 2.32.2 today. But this latest fix still results in the same issue.

Explicitly downgrading to 2.31.0 fixes the issue for now.

Code sample exhibiting the issue

I retested it with the sample code from https://github.com/Azure/azure-iot-sdk-python/blob/main/samples/async-edge-scenarios/invoke_method_on_module.py with same issue.

Console log of the issue

Traceback (most recent call last):
  File "/app/.venv/lib/python3.12/site-packages/requests/adapters.py", line 555, in send
    conn = self.get_connection_with_tls_context(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/requests/adapters.py", line 411, in get_connection_with_tls_context
    conn = self.poolmanager.connection_from_host(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/urllib3/poolmanager.py", line 246, in connection_from_host
    return self.connection_from_context(request_context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/urllib3/poolmanager.py", line 258, in connection_from_context
    raise URLSchemeUnknown(scheme)
urllib3.exceptions.URLSchemeUnknown: Not supported URL scheme http+unix

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/./main.py", line 36, in <module>
    asyncio.run(main())
  File "/usr/local/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/app/./main.py", line 16, in main
    module_client = IoTHubModuleClient.create_from_edge_environment()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/azure/iot/device/aio/patch_documentation.py", line 228, in create_from_edge_environment
    return super(IoTHubModuleClient, cls).create_from_edge_environment(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/azure/iot/device/iothub/abstract_clients.py", line 816, in create_from_edge_environment
    server_verification_cert = hsm.get_certificate()
                               ^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/azure/iot/device/iothub/edge_hsm.py", line 61, in get_certificate
    r = requests.get(
        ^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/requests_unixsocket/__init__.py", line 51, in get
    return request('get', url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/requests_unixsocket/__init__.py", line 46, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/requests/adapters.py", line 559, in send
    raise InvalidURL(e, request=request)
requests.exceptions.InvalidURL: Not supported URL scheme http+unix

[steffenbeermann]

Thank you very much! Had the same issue and could not find the reason. Downgrading the requests solved it for now.

[cartertinney]

Thanks for bringing this to our attention.

The requests and associated urllib dependencies (and their interaction) have often created issues with our package. In this case, they made a security fix that broke another dependency, requests_unixsocket.

Unfortunately, requests_unixsocket does not appear to be actively maintained at this time, so I'm not sure how viable it will be for it to be updated to fix the issue. We will explore some potential solutions here, and if necessary, we will update our library to restrict the dependency to a version that works.

In the meantime (and in general) I would recommend sticking with a specific version of requests and urllib3 that has worked for you prior - this is not the first time such updates have caused breakage, and it likely won't be the last.

[epa095]

@cartertinney : I think it is reasonable to see this as not only an external bug, but also a problem with this projects requirements specification. In setup.py it is specified:

"requests>=2.20.0,<3.0.0",
"requests-unixsocket>=0.1.5,<1.0.0",

which means that this should install cleanly in any environment which satisfy these limits. It is of course impossible to actually guarantee that you can handle all combinations, but its reasonable to expect the dependency requirements to at least be correct for the latest versions you specify for all dependencies. And the fact is that azure-iot-sdk-python is incompatible with requests==2.32.*, and I think that the install_requires should reflect that.

So to sum up; I think you should also change the install_requires here, by either setting "requests>=2.20.0,<2.32.0", or something like "requests>=2.20.0,<3.0.0,!=2.32.*"

[cartertinney]

@epa095

I agree, like I said, above:

We will explore some potential solutions here, and if necessary, we will update our library to restrict the dependency to a version that works.

I'm hesitant to come out and immediately restrict the version because the update to requests in question is a security fix - if we restrict our version in the long term, we're rejecting this security update as well as any future ones. If it comes down to it we will (at least in the short term), but that's not an ideal outcome either.

[fdelu]

There's an issue in the request-unixsocket repo for this issue: msabramo/requests-unixsocket#73

Since the project is abandoned an user forked the project with a fix. Would it be possible to either use that fork or for this SDK to have its own fork? The project is abandoned anyway so it shouldn't matter if the fork doesn't receive any updates other than critical bugs like this one

Edit: just noticed it's your issue @cartertinney. The comment still applies

[cartertinney]

@fdelu

It's something we're considering, certainly, although it's not trivial for us to simply switch to the other fork, as we would need to do a security analysis, etc.

We're discussing our options right now, and will make some decisions on how to proceed in the coming days.

[fdelu]

I understand why just switching to the other fork could have security issues, that's why I suggested doing a separate fork just for this SDK and only fix critical bugs there.

I'll certainly wait for any updates in the next few days, thanks for the info!

[cartertinney]

Update: we will be simply restricting the requests version for the time being (see #1184).

A release will be made including these changes as soon as possible.

[cartertinney]

azure-iot-device 2.13.1 has been released