Adding Transport API support for CONNECT/Upgrade requests.

[tomchristie]

Opening this discussion to give some visibility onto my thought processes on supporting CONNECT/Upgrade requests & how it ties in with the low-level Transport API.

Let's start this off with a recap on sending a request directly using the low-level API.

example.py

import httpx

transport = httpx.HTTPTransport()
r = transport.handle_request(
    method=b"GET",
    url=(b"https", b"example.com", None, b"/"),
    headers=[(b"Host", "example.com")],
    stream=httpx.ByteStream(b""),
    extensions={}
)
print(r)

Shell:

$ python example.py
(200, [(b'Age', b'333103'), (b'Cache-Control', b'max-age=604800'), (b'Content-Type', b'text/html; charset=UTF-8'), (b'Date', b'Mon, 24 May 2021 10:24:47 GMT'), (b'Etag', b'"3147526947+gzip+ident"'), (b'Expires', b'Mon, 31 May 2021 10:24:47 GMT'), (b'Last-Modified', b'Thu, 17 Oct 2019 07:18:26 GMT'), (b'Server', b'ECS (nyb/1D2E)'), (b'Vary', b'Accept-Encoding'), (b'X-Cache', b'HIT'), (b'Content-Length', b'1256')], <httpx._transports.default.ResponseStream object at 0x105a48dd0>, {'http_version': b'HTTP/1.1', 'reason_phrase': b'OK'})

Okay, so next observation here is that one of the sharper things that we'd like to be able to do with httpx is support HTTP requests that use the CONNECT method, or the Upgrade header. See #1150

In both of those two cases, the initial request starts off being negotiated using HTTP, and then switches the mode of connection into a raw read/write stream. This is an important bit of functionality, because it enables use-cases such as WebSockets, which are initially negotiated over HTTP. The sort of thing we'd like to be able to do here (at a low-level) is this kind of thing:

import httpx

transport = httpx.HTTPTransport()
with transport.handle_request(
    method=b"GET",
    url=(b"https", b"example.com", None, b"/"),
    headers=[(b"Host", "example.com"), (b"Connection", b"Upgrade"), (b"Upgrade": b"websocket")],
    stream=httpx.ByteStream(b""),
    extensions={}
) as r:
    status_code, headers, body, extensions = r
    connection = extensions['connection']  # Could be "connection", "socket", "stream", "raw_stream", etc?...
    # Interact with the raw connection stream using `connection.read()`, `connection.write()`, using
    # a library such as `wsproto`.
    # TODO: Expand on this with a real-case example.

One of the neat things we'll be able to provide for with HTTPX here is to ensure that the connection that's returned here is either reading/writing directly to the socket (for HTTP/1.1) or onto a single HTTP stream (for HTTP/2), with either case being handled transparently to the end-user.

Now, something that'd be super wonderful would be if we were able to fully express proxy requests using the Transport API. That'd just really round out the completeness of the API.

We can already support proxy forwarding, because our url format allows the target to be fully specified, so eg to connect to an HTTP URL we can currently do this:

# Proxy forward request...
scheme = proxy_scheme
host = proxy_host
port = proxy_port
target = “http://www.example.com/”
url = (proxy_scheme, proxy_host, proxy_port, target)

r = transport.handle_request(
    method=b"GET",
    url=url,
    headers=[(b"Host", "example.com")],
    stream=httpx.ByteStream(b""),
    extensions={}
)
print(r)

But making a proxy request to an HTTPS URL, requires a CONNECT request.

We'd like to be able to do something like this:

# Proxy tunnel request...
scheme = proxy_scheme
host = proxy_host
port = proxy_port
target = b“www.example.com:443”

with transport.handle_request(
    method=b"CONNECT",
    url=url,
    headers=[(b"Host", "example.com")],
    stream=httpx.ByteStream(b""),
    extensions={}
) as proxy_response:
    status_code, headers, body, extensions = proxy_response
    connection = extensions['connection']
    # We've got a raw connection at this point. First we want to upgrade it to TLS/SSL.
    connection.start_tls(b"www.example.com")
    # Now we want to start sending HTTP over it.
    # In order to do that we'd want to support "connection" as a *request* extension,
    # to give us a way of passing a pre-connected socket-like interface which should be
    # used to read and write the bytes-to-the-wire.
    with transport.handle_request(
        method=b"GET",
        url=url,
        headers=[(b"Host", "example.com")],
        stream=httpx.ByteStream(b""),
        extensions={"connection": connection}
    ) as r:
        ...

What hadn't occurred to me, until I started thinking about proxy CONNECT requests, is that a "connection" extension makes sense both as a response extension, and as a request extension. It'd also be a really neatly powerful bit of kit for us to have, since it provides a window into "read and write the bytes to this thing".

Some slight glossing over here. The Transport API doesn't return context managers. Just a plain-ol-tuple. But trying to get the idea across for now.