Editing Request in trace

[jborean93]

I'm trying to adapt my code to the new way of working introduced with #419. I've got a fairly weird edge case where the authentication protocol I'm using needs access to the TLS context set up during the connection for channel binding. Unfortunately the current auth callback does not fit this model as it is called before the socket is connected and TLS negotiated and the use of connection pools can cause a lot of trickiness when it comes to re-authentication and the like. In the old world I had a very weird setup that relied on private classes in this library which worked for pretty much all the scenarios I needed (Negotiate auth with proxies, socks, etc). The downside to the use of these private interfaces means it doesn't work anymore which is understandable.

When looking at the new way of working I think I've found an acceptable mechanism that uses the new trace functionality that was added. The only downside is I think the way I'm (ab)using it might not be entirely sanctioned and hoping to get some confirmation as to whether this is ok from a public consumption perspective. The POC code I had which demonstrates this in action is the below

Click to expand!

import asyncio
import base64
import types
import typing

import httpcore
import httpx
import spnego
import spnego.channel_bindings
import spnego.tls
from cryptography import x509
from cryptography.exceptions import UnsupportedAlgorithm
from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives import hashes


def get_tls_server_end_point_hash(
    certificate_der: bytes,
) -> bytes:
    """Get Channel Binding hash.

    Get the channel binding tls-server-end-point hash value from the
    certificate passed in.

    Args:
        certificate_der: The X509 DER encoded certificate.

    Returns:
        bytes: The hash value to use for the channel binding token.
    """
    backend = default_backend()

    cert = x509.load_der_x509_certificate(certificate_der, backend)
    try:
        hash_algorithm = cert.signature_hash_algorithm
    except UnsupportedAlgorithm:
        hash_algorithm = None

    # If the cert signature algorithm is unknown, md5, or sha1 then use sha256 otherwise use the signature
    # algorithm of the cert itself.
    if not hash_algorithm or hash_algorithm.name in ["md5", "sha1"]:
        digest = hashes.Hash(hashes.SHA256(), backend)
    else:
        digest = hashes.Hash(hash_algorithm, backend)

    digest.update(certificate_der)
    certificate_hash = digest.finalize()

    return certificate_hash


class AsyncResponseStream(httpx.AsyncByteStream):
    def __init__(self, httpcore_stream: typing.AsyncIterable[bytes]):
        self._httpcore_stream = httpcore_stream

    async def __aiter__(self) -> typing.AsyncIterator[bytes]:
        async for part in self._httpcore_stream:
            yield part

    async def aclose(self) -> None:
        if hasattr(self._httpcore_stream, "aclose"):
            await self._httpcore_stream.aclose()


class AsyncWSManTransport(httpx.AsyncBaseTransport):
    def __init__(self, uri, ssl_context) -> None:
        url = httpx.URL(uri)
        self._auth_complete = False
        self._context = None
        self._hostname_override = url.host
        self._channel_bindings = None

        self._connection = httpcore.AsyncHTTPConnection(
            httpcore.Origin(url.raw_scheme, url.raw_host, url.port),
            ssl_context=ssl_context,
        )

    async def __aenter__(self) -> "AsyncWSManTransport":
        await self._connection.__aenter__()
        return self

    async def __aexit__(
        self,
        exc_type: typing.Type[BaseException] = None,
        exc_value: BaseException = None,
        traceback: types.TracebackType = None,
    ) -> None:
        await self._connection.__aexit__()

    async def handle_async_request(
        self,
        request: httpx.Request,
    ) -> httpx.Response:
        out_token = None

        while True:
            headers = request.headers.raw
            if out_token:
                encoded_token = base64.b64encode(out_token)
                headers.append((b"Authorization", b"Negotiate " + encoded_token))
                out_token = None

            req = httpcore.Request(
                method=request.method,
                url=httpcore.URL(
                    scheme=request.url.raw_scheme,
                    host=request.url.raw_host,
                    port=request.url.port,
                    target=request.url.raw_path,
                ),
                headers=headers,
                content=request.stream,
                extensions=request.extensions,
            )

            resp = await self._connection.handle_async_request(req)

            if self._context:
                for name, value in resp.headers:
                    if name == b"WWW-Authenticate":
                        enc_in_token = value.split(b" ")[1]
                        in_token = base64.b64decode(enc_in_token)
                        out_token = self._context.step(in_token)
                        break

                if not out_token:
                    return httpx.Response(
                        status_code=resp.status,
                        headers=resp.headers,
                        stream=AsyncResponseStream(resp.stream),
                        extensions=resp.extensions,
                    )

                await resp.aread()
                await resp.aclose()

            else:
                return httpx.Response(
                    status_code=resp.status,
                    headers=resp.headers,
                    stream=AsyncResponseStream(resp.stream),
                    extensions=resp.extensions,
                )

    async def trace(
        self,
        event_name: str,
        info: typing.Dict[str, typing.Any],
    ) -> None:
        print(f"{event_name} - {info!s}")

        normalized_name = event_name.lower().replace(".", "_")
        event_handler = getattr(self, f"_{normalized_name}", None)
        if event_handler:
            await event_handler(info)

    async def _http11_send_request_headers_started(self, info: typing.Dict[str, typing.Any]) -> None:
        # The first request needs the context to be set up and the first token added as a header
        if self._context:
            return

        self._context = spnego.client(
            username=None,
            password=None,
            hostname=self._hostname_override,
            channel_bindings=self._channel_bindings,
            service="http",
            context_req=spnego.ContextReq.default,
            options=spnego.NegotiateOptions.none,
            protocol="negotiate",
        )
        token = self._context.step()
        encoded_token = base64.b64encode(token)
        info["request"].headers.append((b"Authorization", b"Negotiate " + encoded_token))

    async def _connection_start_tls_complete(self, info: typing.Dict[str, typing.Any]) -> None:
        # Once the TLS handshake is done we can immediately get the TLS channel bindings used later when creating the
        # auth context (as the headers have started).
        ssl_object = info["return_value"].get_extra_info("ssl_object")
        cert = ssl_object.getpeercert(True)
        cert_hash = get_tls_server_end_point_hash(cert)
        self._channel_bindings = spnego.channel_bindings.GssChannelBindings(
            application_data=b"tls-server-end-point:" + cert_hash
        )


async def connection(uri: str) -> None:
    headers = {
        "Accept-Encoding": "identity",
        "User-Agent": "Python PSRP Client",
    }
    ssl_context = httpx.create_ssl_context(verify=False)
    timeout = httpx.Timeout(30, connect=30, read=30)
    wsman = AsyncWSManTransport(uri, ssl_context)

    async with httpx.AsyncClient(headers=headers, timeout=timeout, transport=wsman) as client:
        resp = await client.post(uri, extensions={"trace": wsman.trace})
        resp.raise_for_status()


asyncio.run(connection("https://server2019.domain.test:5986/wsman"))

Essentially it is creating it's own transport that interacts with AsyncHTTPConnection. On the first request it will send the request with the trace extension and does special actions for the following traces:

• connection.start_tls.complete - Uses the supplied SSL context to get the cert information for the channel binding token
• http11.send_request_headers.started - Sets up the security context with the channel binding token and injects the Authentication header with the first security token returned by the context

Once these weird steps are completed the transport will continue to handle the authentication sequence in handle_async_request as per normal. This works perfectly but I'm worried that injecting this header into the request during the trace is something that could break in the future or shouldn't be done in general.

I've taken the code a bit further where the transport also encrypts the data with the security context when it is completed and all this works wonderfully but this usage is nagging me a bit.

Just as an FYI the next step for me is to try and get Negotiate auth working for proxies which I had working on the old model. I haven't really looked into it in depth but I'm hoping it will be possible. Thanks for the great library, loving the work that you've put into it so far.

[tomchristie]

Heya @jborean93 - this is fantastic love it! 😅

I've got a fairly weird edge case where the authentication protocol I'm using needs access to the TLS context set up during the connection for channel binding.

Right, that makes sense, yup! Complex, but makes sense.

I think I've found an acceptable mechanism that uses the new trace functionality that was added. The only downside is I think the way I'm (ab)using it might not be entirely sanctioned and hoping to get some confirmation as to whether this is ok from a public consumption perspective.

So - I rather think we need some extra docs here. The trace functionality exposes callbacks onto mechanics that from an API perspective is internal implementation detail. Soo.... the upshot is that upgrades which might be considered fine from a SEMVER perspective, might have behaviour changes to what you see in the trace callbacks.

What do we do about that? Well... the sensible thing is to call that out in the docs. Essentially "if you're using the trace extension, then you should avoid relying on the callback information being structured in a particular way. If you are relying on a particular structure then we recommend that you use a strictly pinned version of httpcore, and review any possible changes when upgrading".

Make sense? Would you be up for opening an issue along these lines, citing this discussion?

It's possible that at some point httpcore might have more public API around customising the connections used within the connection pool, but that'd need a lot of thought and refinement before we'd be ready to do that. There might be an alternative pubic API suitable for your use case if we get to that someday.

the next step for me is to try and get Negotiate auth working for proxies

Oh neat - that's pretty interesting. Issue encode/httpx#2033 is somewhat related to that too. The discussion there might help give you some context onto resolving that. Do let us know how you get on!

[jborean93]

Essentially "if you're using the trace extension, then you should avoid relying on the callback information being structured in a particular way
If you are relying on a particular structure then we recommend that you use a strictly pinned version of httpcore, and review any possible changes when upgrading".
Make sense? Would you be up for opening an issue along these lines, citing this discussion?

Totally understand this recommendation but I would be remiss to point out that this is already being used in a somewhat public way with httpx https://github.com/encode/httpx/blob/8dc9b6bd59114439128d23a60609bee09daf508c/httpx/_main.py#L197. I see httpx pins to a version range httpcore>=0.14.0,<0.15.0 but is the plan for httpx to always be reliant on a specific range in the future (once it hits v1?). The mere fact that it has a range means that it could also be susceptible to the same problem I have if httpcore changes the details. My hope was that since httpx does it then it was somewhat public and my concerns were more around the insertion of the auth header with one of the steps not being great.

The concept of pinning is ok, especially in the 0.x days but I do have greater concerns once 1.x starts. Especially since this code is being used in a library so the chances of a dep conflict with users of my library that may also rely on another httpcore version become higher.

'm happy to open an issue to discuss this further but I'm unsure whether this is just for the documentation aspect or to try and find a way I can achieve what I need without relying on implementation details.

There might be an alternative pubic API suitable for your use case if we get to that someday.

Yea I'm really hanging out for the v1.0.0 release to at least have something somewhat stable, I will admit the recent changes made here means I need to rewrite what I had before but that's to be expected from a 0.x release. At the least the changes made so far have reduced the reliance on internal code even if the tracing stuff is implementation details. The reasons why I really want to use httpx/httpcore is that it offers a single API for both sync and asynchronous code. This allows my library to do the same without having different backends for both sides rather than completely different implementations.

Implementing all this myself is a lot of code so I'm trying to avoid as much custom code as I can. Unfortunately my requirements are pretty niche so whatever API I choose will most likely need me plumbing through the depths to get what is needed. Maybe the best option in the end is to just vendor httpcore/httpx and upgrade as new releases come out. Maybe what I will need will become available in a more stable API in the future removing the need to pin all this.

Oh neat - that's pretty interesting. Issue encode/httpx#2033 is somewhat related to that too. The discussion there might help give you some context onto resolving that. Do let us know how you get on!

Will do, I'm sure it will be possible with the custom transport, just need to figure out how the proxies work in the new way. Before I was essentially nesting connections that handle it all which worked beautifully except at that point all the connection classes were private. Now that it's opened up a bit more hopefully it should be less brittle.