How to send an HTTPS query through an HTTPS proxy with aiohttp v3.8?

[webknjaz]

The currently maintained versions of CPython have support for TLS-in-TLS disabled for asyncio. Is there any way to work around this limitation and use HTTPS proxies in aiohttp?

[webknjaz]

The asyncio.loop.start_tls() is available since Python 3.7. But some of the asyncio internals don't set an attribute that would enable creating TLS connections encapsulated inside of other TLS connections (at least, not until Python 3.11).

It is possible to patch the stdlib asyncio to allow this. Add the following monkey-patch in your app and it will work (to the extent of what CPython itself supports):

import asyncio

setattr(asyncio.sslproto._SSLProtocolTransport, "_start_tls_compatible", True)

After this, proceed using the proxy as normal. Here's the doc: https://docs.aiohttp.org/en/stable/client_advanced.html#proxy-support.

Note that this support has only been enabled on the aiohttp side since v3.8.

Refs:

• Add secure proxy support in the client #5992
• https://bugs.python.org/issue37179
• bpo-37179: Support asyncio loop.start_tls() for TLS in TLS python/cpython#28073

⚠️ NOTE: Please, do not file any bugs related to this until CPython starts supporting the underlying feature officially. RELY ON THIS PATCH AT YOUR OWN RISK.

[kumaraditya303]

@webknjaz FYI, This is enabled in 3.11+ and tls in tls works.

[webknjaz]

Thanks for the heads-up. We may need to mention this in the docs and maybe adjust the tests.

[NewUserHa]

Should aiohttp skip this path to avoid conflict on python 3.11?

aiohttp/aiohttp/connector.py

Line 1189 in 1337c45

self._warn_about_tls_in_tls(transport, req)

[Dreamsorcerer]

I assume the check in the method wouldn't trigger in 3.11?

aiohttp/aiohttp/connector.py

Line 992 in 1337c45

if asyncio_supports_tls_in_tls:

[NewUserHa]

ok. I thought that was the temp patch. didn't see that patch needing to be enabled manually.
right, it would not trigger in 3.11.

[danmarinescu]

I see this warning in my logs when making an HTTPs request over an HTTP proxy

/usr/local/lib/python3.10/site-packages/aiohttp/connector.py:905: RuntimeWarning: An HTTPS request is being sent through an HTTPS proxy. This support for TLS in TLS is known to be disabled in the stdlib asyncio. This is why you'll probably see an error in the log below.

Looking at the code it seems that even when using an HTTP proxy aiohttp will send the warning https://github.com/aio-libs/aiohttp/blob/v3.8.3/aiohttp/connector.py#L1249. I find that a bit confusing, given I am not using an HTTPS proxy. Why is this warning logged? Am I missing/misunderstanding something?

[webknjaz]

@danmarinescu I don't remember. Probably start_tls() still gets used. Maybe it's a bug, I'm not sure. But if you end up going down the rabbit hole, make sure to submit a regression test per https://blog.ganssle.io/articles/2021/11/pytest-xfail.html / https://ganssle.io/talks/#xfail-lightning / https://youtu.be/YmAXhlcNbys?t=515.
Also, currently a lot of proxy.py-based tests don't succeed so that would be useful to debug too — so that we could have them enabled w/o an xfail.

[danmarinescu]

Thanks for the reply @webknjaz. At this stage it is unlikely I'll be able to invest the time required (not familiar with the code base) to figure this one out. Sorry 🤷‍♂️

[3awny]

@webknjaz any update on this? Im using python 3.11.2. I'm using a http proxy and still getting warning and error

[MyNameIsDark01]

I'm using python3.11 and the last version of aiohttp btw still having the same warning.

.local/lib/python3.11/site-packages/aiohttp/connector.py:899: RuntimeWarning: An HTTPS request is being sent through an HTTPS proxy. This support for TLS in TLS is known to be disabled in the stdlib asyncio. This is why you'll probably see an error in the log below.

It is possible to enable it via monkeypatching under Python 3.7 or higher. For more details, see:
* https://bugs.python.org/issue37179
* https://github.com/python/cpython/pull/28073

You can temporarily patch this as follows:
* https://docs.aiohttp.org/en/stable/client_advanced.html#proxy-support
* https://github.com/aio-libs/aiohttp/discussions/6044

[webknjaz]

It'd be useful to have more details. Is this a TLS connection over a pure HTTP proxy?

[tiejunhu]

Sometime the underlying transport is not asyncio.sslproto._SSLProtocolTransport and has no "_start_tls_compatible" attribute. So the hack and the check code won't work and warnings is always shown despite it may actually works.

One of the cases is uvloop.loop.TCPTransport

[webknjaz]

Refs:

• incorrect operation with uvloop #7686
• Possible to add _start_tls_compatible attribute to TCPTransport? MagicStack/uvloop#588

A workaround would be to use the built-in Python capability to suppress certain warnings.