Go 1.23: Additional key exchange mechanism X25519Kyber768Draft00 causes AWS Network Firewall to drop packets

[ewbankkit]

v5.65.0 of the Terraform AWS Provider upgraded to Go 1.23.0, which introduced a minor change to the crypto/tls standard library package:

The experimental post-quantum key exchange mechanism X25519Kyber768Draft00 is now enabled by default when Config.CurvePreferences is nil. The default can be reverted by adding tlskyber=0 to the GODEBUG environment variable.

This additional key exchange mechanism causes the length of the TLS ClientHello message to increase, as seen in these packet captures:

v5.64.0 go1.22.6

Frame 30: 344 bytes on wire (2752 bits), 344 bytes captured (2752 bits)
Ethernet II, Src: Apple_42:9a:34 (60:3e:5f:42:9a:34), Dst: Verizon_59:9a:20 (48:5d:36:59:9a:20)
Internet Protocol Version 4, Src: 192.168.1.90, Dst: 52.94.102.126
Transmission Control Protocol, Src Port: 53279, Dst Port: 443, Seq: 1, Ack: 1, Len: 290
Transport Layer Security
    TLSv1.3 Record Layer: Handshake Protocol: Client Hello
        Content Type: Handshake (22)
        Version: TLS 1.0 (0x0301)
        Length: 285
        Handshake Protocol: Client Hello
            Handshake Type: Client Hello (1)
            Length: 281
            Version: TLS 1.2 (0x0303)
            Random: 04961fdeb9d33fffecda028c7bae18c983939ce119313f5f888577d01a93d2fc
            Session ID Length: 32
            Session ID: 974c5f0da41e51fe399c8daee9d50f1c4357dbf715d22539616d22a89962c2f0
            Cipher Suites Length: 28
            Cipher Suites (14 suites)
            Compression Methods Length: 1
            Compression Methods (1 method)
            Extensions Length: 180
            Extension: server_name (len=35) name=sts.ca-central-1.amazonaws.com
            Extension: status_request (len=5)
            Extension: supported_groups (len=10)
            Extension: ec_point_formats (len=2)
            Extension: signature_algorithms (len=26)
            Extension: renegotiation_info (len=1)
            Extension: extended_master_secret (len=0)
            Extension: application_layer_protocol_negotiation (len=14)
            Extension: signed_certificate_timestamp (len=0)
            Extension: supported_versions (len=5) TLS 1.3, TLS 1.2
            Extension: key_share (len=38) x25519
            [JA4: t13d1411h2_cbb2034c60b8_e7c285222651]
            [JA4_r: t13d1411h2_1301,1302,1303,c009,c00a,c012,c013,c014,c02b,c02c,c02f,c030,cca8,cca9_0005,000a,000b,000d,0012,0017,002b,0033,ff01_0804,0403,0807,0805,0806,0401,0501,0601,0503,0603,0201,0203]
            [JA3 Fullstring: 771,49195-49199-49196-49200-52393-52392-49161-49171-49162-49172-49170-4865-4866-4867,0-5-10-11-13-65281-23-16-18-43-51,29-23-24-25,0]
            [JA3: fd75aaca18604d62f2bc8b02b345140f]

v5.66.0 go1.23.0

Frame 148: 104 bytes on wire (832 bits), 104 bytes captured (832 bits)
Ethernet II, Src: Apple_42:9a:34 (60:3e:5f:42:9a:34), Dst: Verizon_59:9a:20 (48:5d:36:59:9a:20)
Internet Protocol Version 4, Src: 192.168.1.90, Dst: 52.94.108.171
Transmission Control Protocol, Src Port: 52515, Dst Port: 443, Seq: 1461, Ack: 1, Len: 50
[2 Reassembled TCP Segments (1510 bytes): #147(1460), #148(50)]
Transport Layer Security
    TLSv1.3 Record Layer: Handshake Protocol: Client Hello
        Content Type: Handshake (22)
        Version: TLS 1.0 (0x0301)
        Length: 1505
        Handshake Protocol: Client Hello
            Handshake Type: Client Hello (1)
            Length: 1501
            Version: TLS 1.2 (0x0303)
            Random: 9c5ad6a75fc2a0dfc40e23384ac1400bbf741c7ac53d1f648597122194cc2556
            Session ID Length: 32
            Session ID: b518747b7be6c9ee34cc3826b13d45ba34a6e25254a1b2e295b4d3aba934319f
            Cipher Suites Length: 26
            Cipher Suites (13 suites)
            Compression Methods Length: 1
            Compression Methods (1 method)
            Extensions Length: 1402
            Extension: server_name (len=35) name=sts.ca-central-1.amazonaws.com
            Extension: ec_point_formats (len=2)
            Extension: renegotiation_info (len=1)
            Extension: extended_master_secret (len=0)
            Extension: signed_certificate_timestamp (len=0)
            Extension: status_request (len=5)
            Extension: supported_groups (len=12)
            Extension: signature_algorithms (len=26)
            Extension: application_layer_protocol_negotiation (len=14)
            Extension: supported_versions (len=5) TLS 1.3, TLS 1.2
            Extension: key_share (len=1258) X25519Kyber768Draft00, x25519
            [JA4: t13d1311h2_f57a46bbacb6_e7c285222651]
            [JA4_r: t13d1311h2_1301,1302,1303,c009,c00a,c013,c014,c02b,c02c,c02f,c030,cca8,cca9_0005,000a,000b,000d,0012,0017,002b,0033,ff01_0804,0403,0807,0805,0806,0401,0501,0601,0503,0603,0201,0203]
            [JA3 Fullstring: 771,49195-49199-49196-49200-52393-52392-49161-49171-49162-49172-4865-4866-4867,0-11-65281-23-18-5-10-13-16-43-51,25497-29-23-24-25,0]
            [JA3: 50982bb7732015b00f10d5024747686f]

The increased message length leads to AWS Network Firewall blocking the message:

{
    "event_timestamp": "1725902638",
    "event": {
        "app_proto": "tls",
        "event_type": "alert",
        "alert": {
            "severity": 3,
            "rev": 0,
            "signature": "",
            "action": "blocked",
            "category": ""
        },
        "proto": "TCP",
        "tls": {
            "version": "UNDETERMINED",
            "ja3": {},
            "ja3s": {}
        },
        "dest_port": 443,
        "timestamp": "2024-09-09T17:23:58.165763+0000"
    }
}

AWS Network Firewall drops the message (causing the TLS handshake to timeout) because its stateful rule capability currently uses Suricata version 6.0.9, and this version of Suricata is known to drop TLS packets beyond a certain length.

v5.67.0 of the Terraform AWS Provider restored use of Go 1.22.6, but we need to have a solution to this problem.

Immediately, the GODEBUG environment variable can be set to tlskyber=0.

Longer-term, our HTTP client can be configured to use the Go 1.22 default key exchange mechanisms (see https://cs.opensource.google/go/go/+/refs/tags/go1.23.1:src/crypto/tls/defaults.go).

An alternative approach is to use recent go.mod enhancements to set default GODEBUG values.
See https://go.dev/doc/godebug#history, golang/go#65573.

Relates #38999.
Relates #39256.
Relates #39110.
Relates #39115.
Relates #39125.
Relates #39141.
Relates #39173.

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[github-actions]

Warning

This issue has been closed, meaning that any additional comments are hard for our team to see. Please assume that the maintainers will not see them.

Ongoing conversations amongst community members are welcome, however, the issue will be locked after 30 days. Moving conversations to another venue, such as the AWS Provider forum, is recommended. If you have additional concerns, please open a new issue, referencing this one where needed.

[github-actions]

This functionality has been released in v5.69.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.