Support for padding extension

[vellrya]

Hello.
I'm using FingerprintClientHello and noticed that it doesn't support padding extension (21), which makes the ja3 fingerprint do not match the original fp in some cases.

There was a proposal in the original library to add support for this extension, but in 3 years it was still not implemented: golang/go#39271

How difficult is it to implement this?

[gaukas]

Padding extension (21) is well implemented and fully supported by (*Fingerprinter).FingerprintClientHello.

I used the following code to establish a connection successfully with a remote server (slightly modified based on TestUTLSHandshakeClientFingerprintedSpecFromRaw.

Code

func TestUTLSHandshakeRaw(t *testing.T) {
        // TLSv1.3 Record Layer: Handshake Protocol: Client Hello
	//     Content Type: Handshake (22)
	//     Version: TLS 1.0 (0x0301)
	//     Length: 512
	// Handshake Protocol: Client Hello
	//     Handshake Type: Client Hello (1)
	//     Length: 508
	//     Version: TLS 1.2 (0x0303)
	//     Random: 7fd76fa530c24816ea9e4a6cf2e939f2350b9486a7bac58e…
	//     Session ID Length: 32
	//     Session ID: d9b01fc4f4b6fe14fe9ce652442d66588d982cb25913d866…
	//     Cipher Suites Length: 36
	//     Cipher Suites (18 suites)
	//         Cipher Suite: TLS_AES_128_GCM_SHA256 (0x1301)
	//         Cipher Suite: TLS_CHACHA20_POLY1305_SHA256 (0x1303)
	//         Cipher Suite: TLS_AES_256_GCM_SHA384 (0x1302)
	//         Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02b)
	//         Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f)
	//         Cipher Suite: TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 (0xcca9)
	//         Cipher Suite: TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (0xcca8)
	//         Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 (0xc02c)
	//         Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (0xc030)
	//         Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA (0xc00a)
	//         Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA (0xc009)
	//         Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (0xc013)
	//         Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (0xc014)
	//         Cipher Suite: TLS_RSA_WITH_AES_128_GCM_SHA256 (0x009c)
	//         Cipher Suite: TLS_RSA_WITH_AES_256_GCM_SHA384 (0x009d)
	//         Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA (0x002f)
	//         Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA (0x0035)
	//         Cipher Suite: TLS_RSA_WITH_3DES_EDE_CBC_SHA (0x000a)
	//     Compression Methods Length: 1
	//     Compression Methods (1 method)
	//     Extensions Length: 399
	//     Extension: server_name (len=34)
	//         Type: server_name (0)
	//         Length: 34
	//         Server Name Indication extension
	//     Extension: extended_master_secret (len=0)
	//         Type: extended_master_secret (23)
	//         Length: 0
	//     Extension: renegotiation_info (len=1)
	//         Type: renegotiation_info (65281)
	//         Length: 1
	//         Renegotiation Info extension
	//     Extension: supported_groups (len=14)
	//         Type: supported_groups (10)
	//         Length: 14
	//         Supported Groups List Length: 12
	//         Supported Groups (6 groups)
	//     Extension: ec_point_formats (len=2)
	//         Type: ec_point_formats (11)
	//         Length: 2
	//         EC point formats Length: 1
	//         Elliptic curves point formats (1)
	//     Extension: application_layer_protocol_negotiation (len=14)
	//         Type: application_layer_protocol_negotiation (16)
	//         Length: 14
	//         ALPN Extension Length: 12
	//         ALPN Protocol
	//     Extension: status_request (len=5)
	//         Type: status_request (5)
	//         Length: 5
	//         Certificate Status Type: OCSP (1)
	//         Responder ID list Length: 0
	//         Request Extensions Length: 0
	//     Extension: key_share (len=107)
	//         Type: key_share (51)
	//         Length: 107
	//         Key Share extension
	//     Extension: supported_versions (len=5)
	//         Type: supported_versions (43)
	//         Length: 5
	//         Supported Versions length: 4
	//         Supported Version: TLS 1.3 (0x0304)
	//         Supported Version: TLS 1.2 (0x0303)
	//     Extension: signature_algorithms (len=24)
	//         Type: signature_algorithms (13)
	//         Length: 24
	//         Signature Hash Algorithms Length: 22
	//         Signature Hash Algorithms (11 algorithms)
	//     Extension: record_size_limit (len=2)
	//         Type: record_size_limit (28)
	//         Length: 2
	//         Record Size Limit: 16385
	//     Extension: padding (len=143)
	//         Type: padding (21)
	//         Length: 143
	//         Padding Data: 000000000000000000000000000000000000000000000000…
	byteString := []byte("1603010200010001fc03037fd76fa530c24816ea9e4a6cf2e939f2350b9486a7bac58ece5753767fb6112420d9b01fc4f4b6fe14fe9ce652442d66588d982cb25913d866348bde54d3899abe0024130113031302c02bc02fcca9cca8c02cc030c00ac009c013c014009c009d002f0035000a0100018f00000022002000001d70656f706c652d70612e636c69656e7473362e676f6f676c652e636f6d00170000ff01000100000a000e000c001d00170018001901000101000b000201000010000e000c02683208687474702f312e310005000501000000000033006b0069001d002065e566ff33dfbeb012e3b13b87d75612bd0fbc3963673df90afed533dccc9b5400170041047fcc2666d04c31272a2e39905c771a89edf5a71dae301ec2fa0e7bc4d0e06580a0d36324e3dc4f29e200a8905badd11c00daf11588977bf501597dac5fdc55bf002b00050403040303000d0018001604030503060308040805080604010501060102030201001c000240010015008f0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
	helloBytes := make([]byte, hex.DecodedLen(len(byteString)))
	_, err := hex.Decode(helloBytes, byteString)
	if err != nil {
		t.Errorf("got error: %v; expected to succeed", err)
		return
	}

	f := &Fingerprinter{}
	generatedSpec, err := f.FingerprintClientHello(helloBytes)
	if err != nil {
		t.Errorf("got error: %v; expected to succeed", err)
	}

	config := Config{ServerName: "gaukas.wang"}
	dialConn, err := net.Dial("tcp", "gaukas.wang:443")
	if err != nil {
		t.Fatalf("failed to dial: %v", err)
	}
	uTlsConn := UClient(dialConn, &config, HelloCustom)
	defer uTlsConn.Close()

	if err := uTlsConn.ApplyPreset(generatedSpec); err != nil {
		t.Errorf("got error: %v; expected to succeed", err)
	}

	err = uTlsConn.Handshake()
	if err != nil {
		t.Errorf("got error: %v; expected to succeed", err)
	}
}

Will you be able to provide a minimal example where you call (*Fingerprinter).FingerprintClientHello with proper padding extension and yield a ClientHelloSpec that is not well padded?

[gaukas]

Also, in terms of the upstream (crypto/tls), they simply don't care.

[vellrya]

Hmm, that's interesting)
Yes, your example actually works, but I have a ClientHello sample where the error is reproduced. Can I send it to the email listed in your profile, as I'm not sure I can publicly disclose the URL of a 3rd party site?

[vellrya]

Original ClientHello:

Fingerptined ClientHello:

The padding seems to have been lost

[gaukas]

Can I send it to the email listed in your profile

That would work.

Please include your original pcap as well as your example code, so I can give it a look. Possibly it is due to how it is parsed/read into your program.

[gaukas]

After inspecting your use case and check with uTLS's implementation, here's what I find:

uTLS uses BoringPaddingStyle, which pads only when the length of a ClientHello is greater than 255 bytes (0xFF):

utls/u_tls_extensions.go

Lines 1052 to 1063 in d39ed1b

	func BoringPaddingStyle(unpaddedLen int) (int, bool) {
	if unpaddedLen > 0xff && unpaddedLen < 0x200 {
	paddingLen := 0x200 - unpaddedLen
	if paddingLen >= 4+1 {
	paddingLen -= 4
	} else {
	paddingLen = 1
	}
	return paddingLen, true
	}
	return 0, false
	}

Your ClientHello is not long enough and according to BoringSSL it should not be padded. If you still want it to be padded, for now temporarily you can manually change the GetPaddingLen to a function that always return 512-length or something. I will try to open a pull request to add an AlwaysPadToXXX to be used for parsed fingerprints since it is better to honor the choice of these input fingerprints even when they are non-standard.