new: Support TLS-PSK (TLS 1.3) (#231)

* uTLS: X25519Kyber768Draft00 hybrid post-quantum key agreement by cloudflare/go (#222)

* crypto/tls: Add hybrid post-quantum key agreement  (#13)

* import: client-side KEM from cloudflare/go

* import: server-side KEM from cloudflare/go

* fix: modify test to get rid of CFEvents.

Note: uTLS does not promise any server-side functionality, and this change is made to be able to conduct unit tests which requires both side to be able to handle KEM Curves.

Co-authored-by: Christopher Wood <caw@heapingbits.net>
Co-Authored-By: Bas Westerbaan <bas@westerbaan.name>

----

Based on:

* crypto/tls: Add hybrid post-quantum key agreement 

Adds X25519Kyber512Draft00, X25519Kyber768Draft00, and
P256Kyber768Draft00 hybrid post-quantum key agreements with temporary
group identifiers.

The hybrid post-quantum key exchanges uses plain X{25519,448} instead
of HPKE, which we assume will be more likely to be adopted. The order
is chosen to match CECPQ2.

Not enabled by default.

Adds CFEvents to detect `HelloRetryRequest`s and to signal which
key agreement was used.

Co-authored-by: Christopher Wood <caw@heapingbits.net>

 [bas, 1.20.1: also adds P256Kyber768Draft00]
 [pwu, 1.20.4: updated circl to v1.3.3, moved code to cfevent.go]

* crypto: add support for CIRCL signature schemes

* only partially port the commit from cloudflare/go. We would stick to the official x509 at the cost of incompatibility.

Co-Authored-By: Bas Westerbaan <bas@westerbaan.name>
Co-Authored-By: Christopher Patton <3453007+cjpatton@users.noreply.github.com>
Co-Authored-By: Peter Wu <peter@lekensteyn.nl>

* crypto/tls: add new X25519Kyber768Draft00 code point

Ported from cloudflare/go to support the upcoming new post-quantum keyshare.

----

* Point tls.X25519Kyber768Draft00 to the new 0x6399 identifier while the
  old 0xfe31 identifier is available as tls.X25519Kyber768Draft00Old.
* Make sure that the kem.PrivateKey can always be mapped to the CurveID
  that was linked to it. This is needed since we now have two ID
  aliasing to the same scheme, and clients need to be able to detect
  whether the key share presented by the server actually matches the key
  share that the client originally sent.
* Update tests, add the new identifier and remove unnecessary code.

Link: https://mailarchive.ietf.org/arch/msg/tls/HAWpNpgptl--UZNSYuvsjB-Pc2k/
Link: https://datatracker.ietf.org/doc/draft-tls-westerbaan-xyber768d00/02/
Co-Authored-By: Peter Wu <peter@lekensteyn.nl>
Co-Authored-By: Bas Westerbaan <bas@westerbaan.name>

---------

Co-authored-by: Bas Westerbaan <bas@westerbaan.name>
Co-authored-by: Christopher Patton <3453007+cjpatton@users.noreply.github.com>
Co-authored-by: Peter Wu <peter@lekensteyn.nl>

* new: enable PQ parrots (#225)

* Redesign KeySharesEcdheParameters into KeySharesParameters which supports multiple types of keys.

* Optimize program logic to prevent using unwanted keys

* new: more parrots and safety update (#227)

* new: PQ and other parrots

Add new preset parrots:
- HelloChrome_114_Padding_PSK_Shuf
- HelloChrome_115_PQ
- HelloChrome_115_PQ_PSK

* new: ShuffleChromeTLSExtensions

Implement a new function `ShuffleChromeTLSExtensions(exts []TLSExtension) []TLSExtension`.

* update: include psk parameter for parrot-related functions

Update following functions' prototype to accept an optional pskExtension (of type *FakePreSharedKeyExtension):
- `UClient(conn net.Conn, config *Config, clientHelloID ClientHelloID)` => `UClient(conn net.Conn, config *Config, clientHelloID ClientHelloID, pskExtension ...*FakePreSharedKeyExtension)`
- `UTLSIdToSpec(id ClientHelloID)` => `UTLSIdToSpec(id ClientHelloID, pskExtension ...*FakePreSharedKeyExtension)`

* new: pre-defined error from UTLSIdToSpec

Update UTLSIdToSpec to return more comprehensive errors by pre-defining them, allowing easier error comparing/unwrapping.

* new: UtlsPreSharedKeyExtension

In `u_pre_shared_key.go`, create `PreSharedKeyExtension` as an interface, with 3 implementations:
- `UtlsPreSharedKeyExtension` implements full support for `pre_shared_key` less resuming after seeing HRR.
- `FakePreSharedKeyExtension` uses CipherSuiteID, SessionSecret and Identities to calculate the corresponding binders and send them, without setting the internal states. Therefore if the server accepts the PSK and tries to resume, the connection fails.
- `HardcodedPreSharedKeyExtension` allows user to hardcode Identities and Binders to be sent in the extension without setting the internal states. Therefore if the server accepts the PSK and tries to resume, the connection fails.

TODO: Only one of FakePreSharedKeyExtension and HardcodedPreSharedKeyExtension should be kept, the other one should be just removed. We still need to learn more of the safety of hardcoding both Identities and Binders without recalculating the latter.

* update: PSK minor changes and example

* Updates PSK implementations for more comprehensible interfaces when applying preset/json/raw fingerprints.
* Revert FakePreSharedKeyExtension to the old implementation. Add binder size checking.
* Implement TLS-PSK example

New bug: setting `tls.Config.ClientSessionCache` will cause PSK to fail. Currently users must set only `tls.UtlsPreSharedKeyExtension.ClientSessionCacheOverride`.

* fix: PSK failing if config session cache set

* Fix a bug causing PSK to fail if Config.ClientSessionCache is set.
* Removed `ClientSessionCacheOverride` from `UtlsPreSharedKeyExtension`. Set the `ClientSessionCache` in `Config`!

Co-Authored-By: zeeker999 <13848632+zeeker999@users.noreply.github.com>

* Optimize tls resumption (#235)

* feat: bug fix and refactor

* feat: improve example docs: add detailed explanation about the design feat: add assertion on uApplyPatch

* fix: address comments
feat: add option `OmitEmptyPsk` and throw error on empty psk by default
feat: revert changes to public interfaces

* fix: weird residue caused by merging conflict

* fix: remove merge conflict residue code

---------

Co-authored-by: Bas Westerbaan <bas@westerbaan.name>
Co-authored-by: Christopher Patton <3453007+cjpatton@users.noreply.github.com>
Co-authored-by: Peter Wu <peter@lekensteyn.nl>
Co-authored-by: zeeker999 <13848632+zeeker999@users.noreply.github.com>
Co-authored-by: 3andne <52860475+3andne@users.noreply.github.com>

common.go

@@ -684,6 +684,14 @@ type Config struct {
 	// This field is ignored when InsecureSkipVerify is true.
 	InsecureSkipTimeVerify bool // [uTLS]
 
+	// OmitEmptyPsk determines whether utls will automatically conceal
+	// the psk extension when it is empty. When the psk extension is empty, the
+	// browser omits it from the client hello. Utls can mimic this behavior,
+	// but it deviates from the provided client hello specification, rendering
+	// it unsuitable as the default behavior. Users have the option to enable
+	// this behavior at their own discretion.
+	OmitEmptyPsk bool // [uTLS]
+
 	// InsecureServerNameToVerify is used to verify the hostname on the returned
 	// certificates. It is intended to use with spoofed ServerName.
 	// If InsecureServerNameToVerify is "*", crypto/tls will do normal
@@ -881,6 +889,7 @@ func (c *Config) Clone() *Config {
 		InsecureSkipVerify:          c.InsecureSkipVerify,
 		InsecureSkipTimeVerify:      c.InsecureSkipTimeVerify,
 		InsecureServerNameToVerify:  c.InsecureServerNameToVerify,
+		OmitEmptyPsk:                c.OmitEmptyPsk,
 		CipherSuites:                c.CipherSuites,
 		PreferServerCipherSuites:    c.PreferServerCipherSuites,
 		SessionTicketsDisabled:      c.SessionTicketsDisabled,

conn.go

@@ -96,7 +96,7 @@ type Conn struct {
 	// clientProtocol is the negotiated ALPN protocol.
 	clientProtocol string
 
-	utls utlsConnExtraFields // [UTLS】 used for extensive things such as ALPS
+	utls utlsConnExtraFields // [UTLS] used for extensive things such as ALPS, PSK, etc
 
 	// input/output
 	in, out   halfConn

examples/tls-resumption/main.go

@@ -0,0 +1,159 @@
+package main
+
+import (
+	"fmt"
+	"net"
+	"strings"
+	"time"
+
+	tls "github.com/refraction-networking/utls"
+)
+
+type ClientSessionCache struct {
+	sessionKeyMap map[string]*tls.ClientSessionState
+}
+
+func NewClientSessionCache() tls.ClientSessionCache {
+	return &ClientSessionCache{
+		sessionKeyMap: make(map[string]*tls.ClientSessionState),
+	}
+}
+
+func (csc *ClientSessionCache) Get(sessionKey string) (session *tls.ClientSessionState, ok bool) {
+	if session, ok = csc.sessionKeyMap[sessionKey]; ok {
+		fmt.Printf("Getting session for %s\n", sessionKey)
+		return session, true
+	}
+	fmt.Printf("Missing session for %s\n", sessionKey)
+	return nil, false
+}
+
+func (csc *ClientSessionCache) Put(sessionKey string, cs *tls.ClientSessionState) {
+	if csc.sessionKeyMap == nil {
+		fmt.Printf("Deleting session for %s\n", sessionKey)
+		delete(csc.sessionKeyMap, sessionKey)
+	} else {
+		fmt.Printf("Putting session for %s\n", sessionKey)
+		csc.sessionKeyMap[sessionKey] = cs
+	}
+}
+
+func runResumptionCheck(helloID tls.ClientHelloID, serverAddr string, retry int, verbose bool) {
+	csc := NewClientSessionCache()
+	tcpConn, err := net.Dial("tcp", serverAddr)
+	if err != nil {
+		panic(err)
+	}
+
+	// Everything below this line is brought to you by uTLS API, enjoy!
+
+	// use chs
+	tlsConn := tls.UClient(tcpConn, &tls.Config{
+		ServerName: strings.Split(serverAddr, ":")[0],
+		// NextProtos:         []string{"h2", "http/1.1"},
+		ClientSessionCache: csc, // set this so session tickets will be saved
+		OmitEmptyPsk:       true,
+	}, helloID)
+
+	// HS
+	err = tlsConn.Handshake()
+	if err != nil {
+		panic(err)
+	}
+
+	var tlsVer uint16
+
+	if tlsConn.ConnectionState().HandshakeComplete {
+		tlsVer = tlsConn.ConnectionState().Version
+		if verbose {
+			fmt.Println("Handshake complete")
+			fmt.Printf("TLS Version: %04x\n", tlsVer)
+		}
+		if tlsVer == tls.VersionTLS13 {
+			if verbose {
+				fmt.Printf("Expecting PSK resumption\n")
+			}
+		} else if tlsVer == tls.VersionTLS12 {
+			if verbose {
+				fmt.Printf("Expecting session ticket resumption\n")
+			}
+		} else {
+			panic("Don't try resumption on old TLS versions")
+		}
+
+		if tlsConn.HandshakeState.State13.UsingPSK {
+			panic("unintended using of PSK happened...")
+		} else if tlsConn.DidTls12Resume() {
+			panic("unintended using of session ticket happened...")
+		} else {
+			if verbose {
+				fmt.Println("First connection, no PSK/session ticket to use.")
+			}
+		}
+
+		tlsConn.SetReadDeadline(time.Now().Add(1 * time.Second))
+		tlsConn.Read(make([]byte, 1024)) // trigger a read so NewSessionTicket gets handled
+	}
+	tlsConn.Close()
+
+	for i := 0; i < retry; i++ {
+		tcpConnPSK, err := net.Dial("tcp", serverAddr)
+		if err != nil {
+			panic(err)
+		}
+
+		tlsConnPSK := tls.UClient(tcpConnPSK, &tls.Config{
+			ServerName:         strings.Split(serverAddr, ":")[0],
+			ClientSessionCache: csc,
+			OmitEmptyPsk:       true,
+		}, helloID)
+
+		// HS
+		err = tlsConnPSK.Handshake()
+		if verbose {
+			fmt.Printf("tlsConnPSK.HandshakeState.Hello.Raw %v\n", tlsConnPSK.HandshakeState.Hello.Raw)
+			fmt.Printf("tlsConnPSK.HandshakeState.Hello.PskIdentities: %v\n", tlsConnPSK.HandshakeState.Hello.PskIdentities)
+		}
+
+		if err != nil {
+			panic(err)
+		}
+
+		if tlsConnPSK.ConnectionState().HandshakeComplete {
+			if verbose {
+				fmt.Println("Handshake complete")
+			}
+			newVer := tlsConnPSK.ConnectionState().Version
+			if verbose {
+				fmt.Printf("TLS Version: %04x\n", newVer)
+			}
+			if newVer != tlsVer {
+				panic("Tls version changed unexpectedly on the second connection")
+			}
+
+			if tlsVer == tls.VersionTLS13 && tlsConnPSK.HandshakeState.State13.UsingPSK {
+				fmt.Println("[PSK used]")
+				return
+			} else if tlsVer == tls.VersionTLS12 && tlsConnPSK.DidTls12Resume() {
+				fmt.Println("[session ticket used]")
+				return
+			}
+		}
+		time.Sleep(700 * time.Millisecond)
+	}
+	panic(fmt.Sprintf("PSK or session ticket not used for a resumption session, server %s, helloID: %s", serverAddr, helloID.Client))
+}
+
+func main() {
+	tls13Url := "www.microsoft.com:443"
+	tls12Url1 := "spocs.getpocket.com:443"
+	tls12Url2 := "marketplace.visualstudio.com:443"
+	runResumptionCheck(tls.HelloChrome_100_PSK, tls13Url, 1, false) // psk + utls
+	runResumptionCheck(tls.HelloGolang, tls13Url, 1, false)         // psk + crypto/tls
+
+	runResumptionCheck(tls.HelloChrome_100_PSK, tls12Url1, 10, false) // session ticket + utls
+	runResumptionCheck(tls.HelloGolang, tls12Url1, 10, false)         // session ticket + crypto/tls
+	runResumptionCheck(tls.HelloChrome_100_PSK, tls12Url2, 10, false) // session ticket + utls
+	runResumptionCheck(tls.HelloGolang, tls12Url2, 10, false)         // session ticket + crypto/tls
+
+}

handshake_client.go

@@ -312,6 +312,12 @@ func (c *Conn) clientHandshake(ctx context.Context) (err error) {
 
 func (c *Conn) loadSession(hello *clientHelloMsg) (
 	session *SessionState, earlySecret, binderKey []byte, err error) {
+	// [UTLS SECTION START]
+	if c.utls.sessionController != nil {
+		c.utls.sessionController.onEnterLoadSessionCheck()
+		defer c.utls.sessionController.onLoadSessionReturn()
+	}
+	// [UTLS SECTION END]
 	if c.config.SessionTicketsDisabled || c.config.ClientSessionCache == nil {
 		return nil, nil, nil, nil
 	}
@@ -450,6 +456,11 @@ func (c *Conn) loadSession(hello *clientHelloMsg) (
 	// Compute the PSK binders. See RFC 8446, Section 4.2.11.2.
 	earlySecret = cipherSuite.extract(session.secret, nil)
 	binderKey = cipherSuite.deriveSecret(earlySecret, resumptionBinderLabel, nil)
+	// [UTLS SECTION START]
+	if c.utls.sessionController != nil && !c.utls.sessionController.shouldLoadSessionWriteBinders() {
+		return
+	}
+	// [UTLS SECTION END]
 	transcript := cipherSuite.hash.New()
 	helloBytes, err := hello.marshalWithoutBinders()
 	if err != nil {
@@ -460,7 +471,6 @@ func (c *Conn) loadSession(hello *clientHelloMsg) (
 	if err := hello.updateBinders(pskBinders); err != nil {
 		return nil, nil, nil, err
 	}
-
 	return
 }
 

handshake_messages.go

@@ -319,7 +319,7 @@ func (m *clientHelloMsg) marshal() ([]byte, error) {
 }
 
 // marshalWithoutBinders returns the ClientHello through the
-// FakePreSharedKeyExtension.identities field, according to RFC 8446, Section
+// PreSharedKeyExtension.identities field, according to RFC 8446, Section
 // 4.2.11.2. Note that m.pskBinders must be set to slices of the correct length.
 func (m *clientHelloMsg) marshalWithoutBinders() ([]byte, error) {
 	bindersLen := 2 // uint16 length prefix

tls_test.go

@@ -854,7 +854,7 @@ func TestCloneNonFuncFields(t *testing.T) {
 			f.Set(reflect.ValueOf("b"))
 		case "ClientAuth":
 			f.Set(reflect.ValueOf(VerifyClientCertIfGiven))
-		case "InsecureSkipVerify", "InsecureSkipTimeVerify", "SessionTicketsDisabled", "DynamicRecordSizingDisabled", "PreferServerCipherSuites":
+		case "InsecureSkipVerify", "InsecureSkipTimeVerify", "SessionTicketsDisabled", "DynamicRecordSizingDisabled", "PreferServerCipherSuites", "OmitEmptyPsk":
 			f.Set(reflect.ValueOf(true))
 		case "InsecureServerNameToVerify":
 			f.Set(reflect.ValueOf("c"))

u_clienthello_json.go

@@ -85,7 +85,9 @@ func (c *CompressionMethodsJSONUnmarshaler) CompressionMethods() []uint8 {
 }
 
 type TLSExtensionsJSONUnmarshaler struct {
-	extensions []TLSExtensionJSON
+	AllowUnknownExt bool // if set, unknown extensions will be added as GenericExtension, without recovering ext payload
+	UseRealPSK      bool // if set, PSK extension will be real PSK extension, otherwise it will be fake PSK extension
+	extensions      []TLSExtensionJSON
 }
 
 func (e *TLSExtensionsJSONUnmarshaler) UnmarshalJSON(jsonStr []byte) error {
@@ -107,14 +109,28 @@ func (e *TLSExtensionsJSONUnmarshaler) UnmarshalJSON(jsonStr []byte) error {
 			// get extension type from ID
 			var ext TLSExtension = ExtensionFromID(extID)
 			if ext == nil {
-				// fallback to generic extension
-				ext = genericExtension(extID, accepter.extNameOnly.Name)
+				if e.AllowUnknownExt {
+					// fallback to generic extension, without recovering ext payload
+					ext = genericExtension(extID, accepter.extNameOnly.Name)
+				} else {
+					return fmt.Errorf("extension %s (%d) is not JSON compatible", accepter.extNameOnly.Name, extID)
+				}
+			}
+
+			switch extID {
+			case extensionPreSharedKey:
+				// PSK extension, need to see if we do real or fake PSK
+				if e.UseRealPSK {
+					ext = &UtlsPreSharedKeyExtension{}
+				} else {
+					ext = &FakePreSharedKeyExtension{}
+				}
 			}
 
 			if extJsonCompatible, ok := ext.(TLSExtensionJSON); ok {
 				exts = append(exts, extJsonCompatible)
 			} else {
-				return fmt.Errorf("extension %d (%s) is not JSON compatible", extID, accepter.extNameOnly.Name)
+				return fmt.Errorf("extension %s (%d) is not JSON compatible", accepter.extNameOnly.Name, extID)
 			}
 		}
 	}