networking: delete std.x; add std.crypto.tls and std.http.Client

[andrewrk]

First, this branch deletes the std.x namespace. I originally agreed to have this "experimental namespace" for @lithdew to play around with networking. Well, playtime is over. I'm working on networking now. So anything valuable that is deleted from std.x needs to get properly upstreamed, by being reintroduced and undergoing code review that was not previously done on the std.x namespace. The null hypothesis is that it's all gone, starting now.

Next, this branch introduces TLS 1.3 support to the Zig standard library. So far, this implementation does not heap allocate under any conditions and it is my goal to keep it that way. This implementation is only just getting started and needs a lot of work until it is ready to be used:

• Rework the parsing logic adding abstractions to generally prevent bugs from happening. Minimize the number of locations that we have to "remember" to check for conditions to prevent buffer overreads. Take advantage of comptime and clean up the code to be more maintainable.
• Remove debug prints
• Send a TLS alert message to the server when an error occurs std.crypto.tls.Client: send an alert message to the server when an error occurs #14167
• Send a notify_close message to the server when the client ends the connection
• Rework the read function to not require a large output buffer.
• verify the server's verify_data in the Finished message
• Add certificate validation support and integration with the operating system to find installed certificates.
• find system-installed root certificates on macOS std.crypto.Certificate.Bundle: find system-installed root certificates on macOS #14169
• find system-installed root certificates on Windows std.crypto.Certificate.Bundle: find system-installed root certificates on Windows #14170
• Process the Server Certificate Verify message
• verify the time duration of certificates
• handle certificate validation chains. For example: A validates the host name, B validates A, Root validates B
• Support more kinds of crypto. Some of the ciphers are still not implemented.
• Do a second pass over RFC 8446 and make sure the implementation is complete and compliant. audit the standard library TLS implementation with respect to RFC 8446 #14178
• implement a TLS server in addition to client so that they can be used to test each other add std.crypto.tls.Server #14171
• more testing against real world servers test std lib TLS implementation against many real world servers #14172
• decide whether we want to support earlier protocols than TLS 1.3. Let's start with TLS 1.3 only.
• make the API support the use case of TLS over an interface other than a networking socket.
• figure out why subdomains aren't working (DNS problem?)
• handle a HelloRetryRequest handshake message
• handle post-handshake messages
• look into TCP fastopen and TCP_QUICKACK look into TCP fastopen and TCP_QUICKACK #14173
• use std.heap.FixedBufferAllocator as a stop-gap for heap allocation within the rsa code borrowed from https://github.com/shiguredo/tls13-zig. @jedisct1 is working on an official std.crypto.rsa implementation which will replace the code.
• send a key update before write_seq overflows it would be impossible for this value to overflow
• handle read_seq overflow as an error
• consider adding a write buffer so that write implementation only makes one call to the underlying write primitive
• come up with a testing plan to get test coverage, particularly on that read function. it will probably involve the TLS server implementation increase test coverage of std.crypto.tls #14174
• additionally verify "key usage" std.crypto.Certificate.verify: additionally verify "key usage" #14175
• better URI parsing API make std.url / std.uri more complete and RFC-compliant #14176
• for wildcard certs, only validate one level of sub-domain std.crypto.Certificate: wildcard certs should only validate one level of sub-domain #14177
• consider shipping root certificates with zig ("dependency zero") ship root SSL certificates along with ziglang.org-vendored tarballs #14168

Finally, this branch adds a very simple HTTP client in order to test drive the TLS implementation.

[Jarred-Sumner]

Very exciting! A few things to look at for future macOS specific optimizations:

• connectx let’s you do connect() and also pass a iovec to send immediately on connection. This could be used for TLS 1.3 0-RTT and and also for http-only as a way to send fire-and-forget http requests with a single syscall when you don’t care about the response body
• For async DNS on macOS, see this https://developer.apple.com/documentation/dnssd/1804700-dnsservicegetaddrinfo?language=objc and https://developer.apple.com/documentation/dnssd/1804698-dnsservicerefsockfd?language=objc

[leap0x7b]

cool certificate bro

[ghost]

I also find this very exciting, as I complained before about the haphazard organization of std.net vs std.x. I think at least these things should get upstreamed into std.net:

• Reader and Writer structs on net.Stream that use the send and recv syscalls, allowing you to pass flags (currently you can only get Reader and Writer that call os.read and os.write, with no flags support)
• A tagged union version of Address?
• A function to parse a string into an address + port

[jorangreef]

CC'ing @guidovranken — we were chatting only yesterday about how to get TLS into Zig (before I saw this!). It would be awesome if we could get this into https://github.com/guidovranken/cryptofuzz for fuzzing.

@andrewrk will this be able to be decoupled from network I/O? For example, so that you can treat it as a state machine to make testing easier, or to integrate with different I/O stacks?

[andrewrk]

Thank you for the introduction. Hi @guidovranken - pleased to meet you. I would be very interested in working with you on getting this implementation properly fuzz tested... after it is ready of course. It is still in an exploratory phase currently.

@andrewrk will this be able to be decoupled from network I/O? For example, so that you can treat it as a state machine to make testing easier, or to integrate with different I/O stacks?

Yes. My current plan is to expose the API in the std.crypto.tls namespace and have it accept a "Context" object, similar to how spawning threads works, and how sorting works. Idea being that you pass in an instance of an object of any type as long as it has the following methods:

const Example = struct {
    // put any fields you want here

    pub const WriteError = error {
        // ...
    };

    pub fn writev(example: *Example, iovecs: []const std.os.iovec_const) WriteError!usize {
        // your implementation here...
    }

    pub const ReadError = error {
        // ...
    };

    pub fn read(example: *Example, buffer: []u8) ReadError!usize {
        // your implementation here...
    }
};

Usage would then look something like this:

var example: Example = .{
    // initialize your object
};
var tls_client = try std.crypto.tls.Client.init(&example, tls_options);
const amt_written = try tls_client.write(&example, bytes_to_send);
const amt_read = try tls_client.read(&example, buffer);

You can imagine how one might build a higher level API on top of this.

I would be interested in working with members of the TigerBeetle team to iterate on this API specifically with regards to integration with your IO_uring abstraction. It is my intention to make the API as simple as possible, while maintaining the property that it can be optimally integrated with any reasonable I/O stack - especially one based on IO_uring as a case study.

[guidovranken]

Sure, I'm now subscribed to this thread and I'll keep an eye on it.

OpenSSL and other projects have a very similar way of fuzzing the TLS stack. The nice thing about this is that the fuzzing corpora will be largely compatible, and we can take the corpus of OpenSSL and other TLS software and use it for the Zig TLS fuzzer, so we instantly have a lot of code coverage.

My Cryptofuzz project actually does not fuzz TLS functions, but only core cryptographic functions (including bignums). This has previously found a few bignum bugs in Zig. I integrated the Cryptofuzz Zig fuzzer into OSS-Fuzz the other day; I will send you any bugs it might find.

Currently the Zig harness for Cryptofuzz only supports bignum ops and HKDF: https://github.com/guidovranken/cryptofuzz/blob/master/modules/zig/cryptofuzz.zig

I work on Cryptofuzz whenever I have time (it's unpaid work apart from commissions). I would like to support more cryptographic primitives like hashing, encryption and elliptic curve ops. If anyone in the Zig ecosystem has time to implement this for Cryptofuzz I'd be happy to get in touch and work with them. This will speed up the development. It's well worth the time investment as the project has found hundreds of bugs in major cryptographic libraries.

Once the TLS library is ready, I propose to make a dedicated zig project on OSS-Fuzz, where we will put the TLS fuzzer, the Cryptofuzz Zig fuzzer, and other stdlib fuzzers.

[nektro]

switch (@intToEnum(HandshakeType, handshake_type) { ?

[andrewrk]

@jedisct1 do I understand correctly that std.crypto does not yet have rsa_pkcs1 implemented?

[jedisct1]

Not yet, this is what I'm working on as we speak.

[andrewrk]

Wonderful! I'll move on to other tasks besides certificate validation for now, then. I hope you are avoiding heap allocation in the implementation. We can always know the upper bound of calculations since the bit width is fixed (i.e. 1024, 2048, 4096). If you wanted to experiment with literally using i1024, i2048, i4096 integer types, let me know and I will start working on the compiler improvements to ensure that the required operations (multiplication, division, etc.) will be lowered properly on such large types. Currently, beyond i128, one will get a crash in LLVM lowering code for some operations.

For what it's worth, I noticed on jedisct1/libsodium#368 that it would be beneficial to try to avoid RSA. I tried asking google.com for a non-RSA signature like this:

--- a/lib/std/crypto/tls/Client.zig
+++ b/lib/std/crypto/tls/Client.zig
@@ -53,15 +53,9 @@ pub fn init(stream: net.Stream, ca_bundle: crypto.CertificateBundle, host: []con
         0x02, // byte length of supported versions
         0x03, 0x04, // TLS 1.3
     }) ++ tls.extension(.signature_algorithms, enum_array(tls.SignatureScheme, &.{
-        .rsa_pkcs1_sha256,
-        .rsa_pkcs1_sha384,
-        .rsa_pkcs1_sha512,
         .ecdsa_secp256r1_sha256,
         .ecdsa_secp384r1_sha384,
         .ecdsa_secp521r1_sha512,
-        .rsa_pss_rsae_sha256,
-        .rsa_pss_rsae_sha384,
-        .rsa_pss_rsae_sha512,
         .ed25519,
     })) ++ tls.extension(.supported_groups, enum_array(tls.NamedGroup, &.{

but it sent RSA-signed certificates regardless.

[jedisct1]

@andrewrk RSA keys and signatures are big, it can be tricky to us and implement safely, and there are usually better alternatives.

However, for TLS certs, if someone decided to only sign their cert with RSA, there's not much we can do besides comply and use RSA to verify them.

RSA also still has some unique properties that are useful in modern protocols (for example, see blind RSA signatures that I'm an co-author of -- It has a Zig implementation that currently depends on OpenSSL/BoringSSL which is sad).

For native integers > 128 bits, jedisct1@36833db has been sitting in my tree for a while (which, at least, allows formatting large numbers), but I forgot to turn it into a PR.