Server side ALPN support for SslStream on Mac

[wfurt]

This is follow-up on #26534

SSL API on OSX/MacOS does not provide sufficient functionality to implement server side ALPN.
That is primarily missing code to parse client hello in SSLProcessClientHelloExtensions() and callback (or some other mechanism) for server to choose from provided list.

[karelz]

Thanks @wfurt. Did you get chance to contact Apple and make them aware of this gap? Is there a link to the case?

[karelz]

@wfurt ping

[LouisStAmour]

It’s possible this is resolved in other networking APIs? https://forums.developer.apple.com/thread/116193

[wfurt]

I could not find any reasonable documentation. From the post, I'm not sure how the negotiation would work.
It would be great if somebody can put together simple C example.

cc: @filipnavara @Tratcher

[LouisStAmour]

Did you try the C nwcat sample referenced at the bottom of that thread? You can download the sample at https://developer.apple.com/documentation/network/implementing_netcat_with_network_framework and tweak it following that post’s instructions?

It’s sparsely documented, but there are C language symbols and searching for these online should reveal other samples on Github, etc. https://developer.apple.com/documentation/network/c-language_symbols

To clarify, given deprecation of other networking stacks it’s extremely unlikely Apple will do anything more than bug fixes to the existing APIs other than Network.framework. I’d suggest a third-party SSL library if you can’t get Network.framework going. :)

There’s a video describing the framework here: https://developer.apple.com/videos/play/wwdc2018/715/

There should also be documented C headers floating around in SDKs or you can find them copied online in places.

[karelz]

There is new API on Mac Catalina that we should explore if it helps.

[daveyostcom]

There is new API on Mac Catalina that we should explore if it helps.

fails on Catalina, final release 10.15 (19A583)
dotnet 3.0.100

0 Thu 14:41:13 /tmp
207 Z% dotnet new grpc -o GrpcGreeter
cd GrpcGreeter
dotnet run
Getting ready...
The template "ASP.NET Core gRPC Service" was created successfully.

Processing post-creation actions...
Running 'dotnet restore' on GrpcGreeter/GrpcGreeter.csproj...
  Restore completed in 2.49 sec for /private/tmp/GrpcGreeter/GrpcGreeter.csproj.

Restore succeeded.

warn: Microsoft.AspNetCore.Server.Kestrel[0]
      Unable to bind to https://localhost:5001 on the IPv4 loopback interface: 'HTTP/2 over TLS is not supported on macOS due to missing ALPN support.'.
warn: Microsoft.AspNetCore.Server.Kestrel[0]
      Unable to bind to https://localhost:5001 on the IPv6 loopback interface: 'HTTP/2 over TLS is not supported on macOS due to missing ALPN support.'.
crit: Microsoft.AspNetCore.Server.Kestrel[0]
      Unable to start Kestrel.
System.IO.IOException: Failed to bind to address https://localhost:5001.
 ---> System.AggregateException: One or more errors occurred. (HTTP/2 over TLS is not supported on macOS due to missing ALPN support.) (HTTP/2 over TLS is not supported on macOS due to missing ALPN support.)
 ---> System.NotSupportedException: HTTP/2 over TLS is not supported on macOS due to missing ALPN support.
   at Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware..ctor(ConnectionDelegate next, HttpsConnectionAdapterOptions options, ILoggerFactory loggerFactory)
   at Microsoft.AspNetCore.Hosting.ListenOptionsHttpsExtensions.<>c__DisplayClass12_0.<UseHttps>b__0(ConnectionDelegate next)
   at Microsoft.AspNetCore.Server.Kestrel.Core.ListenOptions.Build()
   at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServer.<>c__DisplayClass21_0`1.<<StartAsync>g__OnBind|0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindEndpointAsync(ListenOptions endpoint, AddressBindContext context)
   at Microsoft.AspNetCore.Server.Kestrel.Core.LocalhostListenOptions.BindAsync(AddressBindContext context)
   --- End of inner exception stack trace ---
 ---> (Inner Exception dotnet/corefx#1) System.NotSupportedException: HTTP/2 over TLS is not supported on macOS due to missing ALPN support.
   at Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware..ctor(ConnectionDelegate next, HttpsConnectionAdapterOptions options, ILoggerFactory loggerFactory)
   at Microsoft.AspNetCore.Hosting.ListenOptionsHttpsExtensions.<>c__DisplayClass12_0.<UseHttps>b__0(ConnectionDelegate next)
   at Microsoft.AspNetCore.Server.Kestrel.Core.ListenOptions.Build()
   at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServer.<>c__DisplayClass21_0`1.<<StartAsync>g__OnBind|0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindEndpointAsync(ListenOptions endpoint, AddressBindContext context)
   at Microsoft.AspNetCore.Server.Kestrel.Core.LocalhostListenOptions.BindAsync(AddressBindContext context)<---

   --- End of inner exception stack trace ---
   at Microsoft.AspNetCore.Server.Kestrel.Core.LocalhostListenOptions.BindAsync(AddressBindContext context)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.AddressesStrategy.BindAsync(AddressBindContext context)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindAsync(IServerAddressesFeature addresses, KestrelServerOptions serverOptions, ILogger logger, Func`2 createBinding)
   at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServer.StartAsync[TContext](IHttpApplication`1 application, CancellationToken cancellationToken)
Unhandled exception. System.IO.IOException: Failed to bind to address https://localhost:5001.
 ---> System.AggregateException: One or more errors occurred. (HTTP/2 over TLS is not supported on macOS due to missing ALPN support.) (HTTP/2 over TLS is not supported on macOS due to missing ALPN support.)
 ---> System.NotSupportedException: HTTP/2 over TLS is not supported on macOS due to missing ALPN support.
   at Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware..ctor(ConnectionDelegate next, HttpsConnectionAdapterOptions options, ILoggerFactory loggerFactory)
   at Microsoft.AspNetCore.Hosting.ListenOptionsHttpsExtensions.<>c__DisplayClass12_0.<UseHttps>b__0(ConnectionDelegate next)
   at Microsoft.AspNetCore.Server.Kestrel.Core.ListenOptions.Build()
   at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServer.<>c__DisplayClass21_0`1.<<StartAsync>g__OnBind|0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindEndpointAsync(ListenOptions endpoint, AddressBindContext context)
   at Microsoft.AspNetCore.Server.Kestrel.Core.LocalhostListenOptions.BindAsync(AddressBindContext context)
   --- End of inner exception stack trace ---
 ---> (Inner Exception dotnet/corefx#1) System.NotSupportedException: HTTP/2 over TLS is not supported on macOS due to missing ALPN support.
   at Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware..ctor(ConnectionDelegate next, HttpsConnectionAdapterOptions options, ILoggerFactory loggerFactory)
   at Microsoft.AspNetCore.Hosting.ListenOptionsHttpsExtensions.<>c__DisplayClass12_0.<UseHttps>b__0(ConnectionDelegate next)
   at Microsoft.AspNetCore.Server.Kestrel.Core.ListenOptions.Build()
   at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServer.<>c__DisplayClass21_0`1.<<StartAsync>g__OnBind|0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindEndpointAsync(ListenOptions endpoint, AddressBindContext context)
   at Microsoft.AspNetCore.Server.Kestrel.Core.LocalhostListenOptions.BindAsync(AddressBindContext context)<---

   --- End of inner exception stack trace ---
   at Microsoft.AspNetCore.Server.Kestrel.Core.LocalhostListenOptions.BindAsync(AddressBindContext context)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.AddressesStrategy.BindAsync(AddressBindContext context)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindAsync(IServerAddressesFeature addresses, KestrelServerOptions serverOptions, ILogger logger, Func`2 createBinding)
   at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServer.StartAsync[TContext](IHttpApplication`1 application, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Hosting.GenericWebHostService.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
   at GrpcGreeter.Program.Main(String[] args) in /private/tmp/GrpcGreeter/Program.cs:line 15
134 Thu 14:41:46 /tmp/GrpcGreeter
208 Z% 

[JamesNK]

I believe that is an explicit error designed to provide feedback about the problem. TLS + HTTP/2 + macOS = error. TLS is not being attempted.

To properly test you would need to disable the check.

[Tratcher]

Failure is expected at the moment, SslStream as not been updated yet to call the new macOS API.

[wfurt]

At this point it is not even clear if use of new API would be feasible. as @Tratcher mentioned, there is no change until SslStream (and this issues) is fixed.

[bancek]

Would it be possible to use OpenSSL on MacOS instead of Network.framework? OpenSSL is already used for Linux so the implementation could be reused. Maybe offer an alternative build.

[wfurt]

We used to do that. But that brings another set of issues - like lack of OS integration for certificate management. That seems more important than server ALPN. If this is critical for somebody you can always run it in Linux container with openssl.

[meteatamel]

I also ran into this issue. This means that the gRPC template/sample does not work out of the box on Mac?

[Tratcher]

I also ran into this issue. This means that the gRPC template/sample does not work out of the box on Mac?

Correct, you'll get a specific exception to that effect.

[dvtkrlbs]

Are there any progress on this?

[wfurt]

not at the moment. This may get to 6.0 but it requires to re-write PAL to use new API so it is significant chunk of work.

[JamesNK]

Are there are plans to investigate this in .NET 6? No TLS on macOS is a point of friction, particularly for gRPC apps.

[wfurt]

It is low on team priority @JamesNK so I would not put much hope in 6.0. The effort is significant and resources sparse. If you think this is important convince stakeholders.

[filipnavara]

I did some investigation into this when doing the iOS TLS bring-up work.

The current TLS implementation uses Security.framework which is deprecated. Underneath the Security.framework with public API there's a coreTLS library with private API. Both of them are open source and the Security APIs are deprecated. In theory there's some ALPN support in there, even for the server side, but in practice it's broken on the coreTLS layer. The code parsing Client Hello messages is missing ALPN code path which in turns prevents getting the peer's ALPN data. Thanks to @wfurt's earlier efforts .NET has managed TLS frame parser that could be used to retrieve the ALPN data and then select the correct protocols to send back in the Server Hello. I successfully got that part running locally. Unfortunately coreTLS is unable to send the ALPN data back either. When coreTLS encodes the Server Hello packet it checks the alpn_announced flag which is never set since it didn't try to parse the ALPN from the client packet. Hence it's irreparably broken on the Apple API side.

The newer Network.framework APIs are considerably more high level. One basically tells the framework how to stack the protocols and then make the connection. It is thus impossible to use TLS without a transport protocol such as TCP, UDP or domain sockets. There are some APIs to set the transport protocol but I was unable to make it work with any custom protocol. In fact, there's no way to create a custom protocol except for a "framer" one which is designed to frame application-level protocols and not transport level ones. Other frameworks (Python, Ruby, QT) which implement similar abstraction as SslStream came to the conclusion that it's basically impossible to mimic it using the new Network.framework APIs (1 2 3 4).

It's possible to use the "framer" custom protocol to high jack the TLS traffic before it reaches the transport protocol. Thus, in theory it's possible to create a dummy transport (eg. local domain socket) and get the Network framework to do TLS handshake on top of it and intercept all the traffic. Trivial example is here. This is reasonably doable for the client-side flows. I was not able to achieve the same thing for the server-side handshake.

Funnily enough, Apple are competing with themselves by maintaining the SwiftNIO framework that has its own TLS stack based on BoringSSL. 🤦🏻‍♂️

UPD: For completeness, Network framework also uses BoringSSL to implement the TLS layer.

cc @mandel-macaque

[wfurt]

thanks for digging and good summary @filipnavara. The findings on Security.framework match my with my memory.

[chrisdrobison]

Any progress on this? This issue makes development of GRPC on Mac where you need GRPC web pretty much unusable. As I've discovered in other threads, that's because browsers do not implement h2c.

[wfurt]

Not really. It is on 7.x list but as @filipnavara and other look into the integration will be probably difficult (or impossible) because of structural differences.
Would H/3 be possibility for you @chrisdrobison? While you would need to jump through some hoops, it should be possible and I'm reasonably confident we can fix possible lingering issues.
Since we did integration with OpenSSL for QUIC, we could possibly have alternative mode where the TLS is powered by OpenSSL instead of native Apple's stack. While there are some security implications we could possibly get on par with Linux. That may be OK for development.

[chrisdrobison]

H/3 isn’t an option for me right now. I’ll experiment with the VS code remote container stuff and see if that will suffice for now.

[chrisdrobison]

Yeah, it looks like the dev container capabilities of VS Code will work well and do what I need for local development. Seems to work pretty well.

[razfriman]

Is there any roadmap update to when this might happen?

Local dev of gRPC and MacOS is pretty sub-par with the only workaround currently to manually disable security features and to toggle this code snippet with directives.

Would love to see 1st class support for this experience while programming on Mac OS

[davidfowl]

Agree, we need to revisit the plan here, even if it's just for development purposes.

cc @JamesNK

[wfurt]

The consensus so far is that it is not possible with available macOS API. That leaves IMHO alternative with OpenSSL backend or HTTP/3. (that gives gRPC without dependency on SslStream)

[TyBarthel]

Are there any disadvantages to advising macOS users to use NGINX to handle SSL and grpc_set_header to send along cert and other details if needed in application code?

[chrisdrobison]

I have no idea what the workload would be like, but would this be something worth making a Microsoft specific fork of OpenSSL for so support could be added? Google forked OpenSSL for Chrome to make the kinds of changes they needed so this kind of thing would not be unique.

[davidfowl]

The consensus so far is that it is not possible with available macOS API. That leaves IMHO alternative with OpenSSL backend or HTTP/3. (that gives gRPC without dependency on SslStream)

HTTP/2 not HTTP/3. I'm thinking OpenSSL on MacOS supported for development scenarios only for gRPC as a way to scope the work/support.

[Leonardo-Ferreira]

do we have any updates here? .net 7 is right around the corner and as a Mac user, I would surely love to remove the #if DEBUGMAC from my code AND let's remember that advanced troubleshooting is really really hard

[davidfowl]

.NET 7 has changed nothing, we're making plans for .NET 8 to solve this.

[wfurt]

triage: we should investigate for 8.0

[wfurt]

I put out fix for 8.0. It would be great if people can give it try and provide feedback once the daily build is available. My testing is somewhat limited to runtime and machines I have access to.
cc: @Tratcher @JamesNK

[Tratcher]

I've filed dotnet/aspnetcore#45569 to track this on our side.

[davidfowl]

Very cool @wfurt !