Exploration of OpenSSL support in the TLS security transport

[MichelSantos]

Response to the issue "IPFS Bounty: Explore OpenSSL support in the TLS security transport"

[MichelSantos]

Executive Summary

This pull request (PR) responds to the issue "IPFS Bounty: Explore OpenSSL support in the TLS security transport".

The pull request retains the existing implementation of Go's standard TLS transport code (StdTLS). Its implementation class, called Transport, was renamed to StdTLSTransport to distinguish it from the new Open SSL implementation (OpenSSL) called OpenSSLTransport. A new interface called IDTransport was created to ease the re-use of existing units tests with either implementation. New benchmarks were created to collect the metrics requested by the issue.

The original standard TLS and the new Open SSL implementations from this PR were tested on a Raspberry Pi 4 and a multi-processor Intel Core i5 platform. The benchmarking results show that the Open SSL implementation is slower than the standard TLS implementation on both platforms. The extra time is predominantly consumed during invocations of the Open SSL C functions in the Open SSL library on the operating system.

Implementation Overview

The libp2p-tls module provides an instance of a SecureTransport, which is defined in go-libp2p-core/sec. This interface exposes the SecureInbound() and SecureOutbound() functions for use by other libp2p modules.

The standard TLS transport (StdTLS) is one implementation of the SecureTransport interface and makes use of the standard net and crypto packages from Go to provides a secure transport class implementing the TLS 1.3 protocol. This implementation has not been changed except to rename the class from Transport to StdTLSTransport.

The new Open SSL transport (OpenSSL) called OpenSSLTransport also honors the SecureTransport interface but is implemented by layering itself on the go-openssl v0.0.7 module which is further layered on the operating system's Open SSL library.

Implementation Deficiencies

TLS Protocol Downgrade

This OpenSSL implementation is likely susceptible to a TLS downgrade from the default usage of TLS 1.3 which is a violation of the libp2p TLS specification of "The libp2p handshake uses TLS 1.3 (and higher). Endpoints MUST NOT negotiate lower TLS versions." The violation could occur if the remote peer proposes a maximum protocol version below TLS 1.3 during the TLS protocol handshake. The deficiency is caused by the absence in the go-openssl v0.0.7 API of the ability to set a minimum TLS protocol version.

This implementation can be trivially fixed after go-openssl is updated to permit the setting.

Detection of a Multi-Certificate Chain from a Remote Peer

This OpenSSL implementation fails to honor the libp2p TLS specification of "Endpoints MUST abort the connection attempt if more than one certificate is received ...". The deficiency is caused by the absence in the go-openssl v0.0.7 API of the ability to inspect the remote peer's certificate chain length.

This implementation can be trivially fixed after go-openssl is updated to permit the inspection.

Tests

The existing tests were re-organized to make use of the new IDTransport interface so that they could be used with either implementation. The tests were further adapted to exercise different combinations of "server" and "client" peers using either the StdTLS or the OpenSSL transport. A "server" was defined as a libp2p peer listening for inbound connections from any other libp2p peer. A "client" was defined as a libp2p peer initiating an outbound connection to a libp2p peer that was expected to bear a peer ID already known to the client. (The client's foreknowledge of the server's peer ID is outside the scope of this module and issue.)

The existing 20 tests between a StdTLS server and a StdTLS client (StdTLSServer-StdTLSClient) were adapted to cover

• an OpenSSL server with an StdTLS client (OpenSSLServer-StdTLSClient)
• a StdTLS server with an OpenSSL client (StdTLSServer-OpenSSLClient)
• an OpenSSL server with an OpenSSL client (OpenSSLServer-OpenSSLClient)

resulting in 58 additional tests. Two tests have been squashed as described in the "Disabled Tests" section.

Adaptations

Adaptations were required for the detection of invalid certificate tests sent by a remote libp2p peer. The adaptations were necessary because the go-openssl v0.0.7 module, which uses the available Open SSL library, behaves differently than the StdTLS implementation. Some of the differences are in the text of the error messages. Other differences relate to the go-openssl module not permitting the sending of optional TLS alert codes (which appear as "SHOULD send an appropriate fatal alert" rather than "MUST send" in Section 6.2 of TLS 1.3 RFC8446). Consequently, validation errors that are detected during the security transport handshake manifest themselves in different ways than when an OpenSSL peer is involved versus when a pair of StdTLS peers are involved. Therefore, the tests involving OpenSSL peers were modified to expect alternate errors.

This implementation can be updated to produce errors messages that are more similar to those produced with the StdTLS transport after go-openssl is updated to permit the transmission of discretionary alert codes to the remote peer.

Test Deficiencies

One of the invalid certificate tests called "certificate chain contains 2 certs" (or "twoCerts" for short) was not able to be reproduced in its entirety. The desired behavior of the test is for a peer to validate the certificate presented by a newly connecting and unknown peer and to then reject the peer's certificate because the certificate chain contains more than a single valid certificate.

Imperfect Tests

The current version of the "twoCerts" invalid certificate test does pass with OpenSSL peer as either a server or a client. The success of this test relies on the detection of the inability to establish a secure connection. While the failure is correctly identified, the failure is not explicitly a rejection of the presence of two certificates. Rather the rejection is caused by the second/bottom-certificate in the chain not being consistent with the certificate's overall signature. In other words, the test correctly detects that the certificate is invalid but not for the reason intended by the original author of the test.

To emphasize this point, it is possible for a two-certificate chain to be accepted by an OpenSSL peer, that is using go-openssl API v0.0.7 and below, under certain conditions. For example, a remote peer presenting a certificate with the following characteristics to an OpenSSL peer will not be rejected: the certificate chain consists of two identical certificates each with a properly signed public key as an extension field, and where the entire certificate is signed by the corresponding private key.

A proper rejection by an OpenSSL peer of a remote peer's multi-certificate chain requires a change to the go-openssl API v0.0.7 that will permit the inspection of the remote peer's certificate chain's length.

These tests can be improved simulatenously with the resolution of the Detection of a Multi-Certificate Chain from a Remote Peer deficiency.

Disabled Tests

The current version of the "twoCerts" invalid certificate test does NOT pass with an OpenSSL server and OpenSSL client because the remote peer accepts the two-certificate chain without any problems. This is due to the fundamental problem described in the Detection of a Multi-Certificate Chain from a Remote Peer.

On a positive note, the usage of go-openssl does correctly verify the first certificate on the chain as being valid and consistent with the signature on the entire certificate. However, as the spirit of the test seeks to reject such a certificate regardless of the validity of the first certificate on the chain per libp2p TLS specification, these two tests (one belonging to when the invalid certificate chain is presented by a client, and one when when it is presented by the server) have been disabled.

These tests can be trivially re-enabled simulatenously with the resolution of the Detection of a Multi-Certificate Chain from a Remote Peer deficiency.

Benchmarks

Benchmarks were created to exercise different combinations of "server" and "client" peers using either the StdTLS or the OpenSSL transports. The definitions of server and client can be found in the Tests section. The benchmarks situate both the server and the client on the same platform which eliminates network congestion as a confounding factor in the results.

Four categories of benchmarks were used to address:

• a StdTLS server listening for connections from StdTLS clients (StdTLSServer-StdTLSClient)
• an OpenSSL server listening for connections from StdTLS clients (OpenSSLServer-StdTLSClient)
• a StdTLS server listening for connections from OpenSSL clients (StdTLSServer-OpenSSLClient)
• an OpenSSL server listening for connections from OpenSSL clients (OpenSSLServer-OpenSSLClient)

Within each category, three benchmarks were measured: connection throughput, handshake throughput, and ping latency (a.k.a. connection latency).

The connection throughput benchmark measures the average duration to (1) create an insecure TCP connection, (2) secure the connection (with a security transport handshake), and (3) close the connection.

The handshake throughput benchmark isolates the measurement to the portion of the connection establishment related to securing an insecure connection (which is Part 2 of the connection throughput benchmark). It measures the average duration of the security transport's handshake. During the handshake, (a) the client cryptographically verifies the servers' certificates to match the identification expected by the client, and (b) the server determines the client's peer identification as cryptographically derived from certificate presented by the client.

Ping latency measures the average duration of sending six bytes between two peers over an established, secured connection.

Execution

The new benchmarks were written to be used with Go's standard benchmarking capability. The benchmarks were measured on a Raspberry Pi 4 ARM64 platform, and an Intel multi-processor AMD64 platform. The benchmarks were performed on both platforms with go1.15.5. The execution was configured with the following switches:

• a benchtime of 5 seconds (--benchtime=5s)
• a single CPU (-cpu=1)
• a build tag to include the new OpenSSL transport, tests, and benchmarks (-tags=openssl)

The typical command line usage was

go test -bench=. --benchtime=5s -cpu=1 -tags=openssl

Benchmarking Results

Benchmarks were collected on two separate platforms: a Raspberry Pi 4 Model B platform (RPi4), and a multi-processor Intel Core i5 (i5). The benchmarks were further collected under different combinations of the server and client running either the StdTLS transport or the OpenSSL transport. The raw benchmarking results can be found in the Appendix.

Benchmark	Handshake LatencyA [ns/op]	Handshake ThroughputB per second	Connection LatencyC [ns/op]	Connection ThroughputD per second
RPi4 StdTLSServer-StdTLSClient	5,783,148	173	111,170	118
RPi4 StdTLSServer-OpenSSLClient	7,609,952	131	145,048	99
RPi4 OpenSSLServer-StdTLSClient	7,052,636	142	144,747	93
RPi4 OpenSSLServer-OpenSSLClient	9,101,175	110	175,600	85
i5 StdTLSServer-StdTLSClient	729,912	1,370	9,053	867
i5 StdTLSServer-OpenSSLClient	1,375,445	727	16,116	627
i5 OpenSSLServer-StdTLSClient	1,164,155	859	16,001	584
i5 OpenSSLServer-OpenSSLClient	1,853,212	540	20,597	471

Note A: The handshake latency correspond to the BenchmarkHandshake metrics from the Appendix.

Note B: The handshake throughput is calculated as the inverse of the shown handshake latency.

Note C: The connection latency is defined as the "ping latency" metric or BenchmarkLatency metrics from the Appendix.

Note D: The connection throughput is calculated as the inverse of the average connection duration BenchmarkConnections metrics from the Appendix.

Durations Relative to the StdTLS Server-StdTLS Client pairing

The results of the different peer-pairs are compared by measuring their performance relative to StdTLS Server-StdTLS Client pairing.

Peer-Pair	Connection Duration	Handshake Duration	Ping Latency Duration
StdTLSServer-OpenSSLClient on RPi	20% slower	32% slower	30% slower
OpenSSLServer-StdTLSClient on RPi	27% slower	22% slower	30% slower
OpenSSLServer-OpenSSLClient on RPi	39% slower	57% slower	58% slower
StdTLSServer-OpenSSLClient on i5	38% slower	88% slower	78% slower
OpenSSLServer-StdTLSClient on i5	48% slower	59% slower	77% slower
OpenSSLServer-OpenSSLClient on i5	84% slower	154% slower	128% slower

The results demonstrate that the usage of the OpenSSL transport by either peer always results in longer durations than when the StdTLS transport is used by both peers. The duration is approximately doubled when both the server and client peers use the OpenSSL transport.

Significance of the Handshake

The duration of the handshake is compared to the duration of raising and tearing down the connection.

Peer-Pair	Duration of the Handshake Relative to the Connection on RPi4	Duration of the Handshake Relative to the Connection on i5
StdTLSServer-StdTLSClient	68%	63%
StdTLSServer-OpenSSLClient	75%	86%
OpenSSLServer-StdTLSClient	66%	68%
OpenSSLServer-OpenSSLClient	78%	87%

These results show that 63% to 87% of the connection duration is spent on the handshake. They also show that the relative duration of the handshake is noticeably larger when the client uses the OpenSSL transport from this implementation.

Profiler Analysis

The benchmarking results from both platforms demonstrate that the usage of the OpenSSL transport from this implementation is always slower than the StdTLS transport. The cause of this sluggishness was investigated by collecting profiling metrics with Go's -cpuprofile profile switch. These metrics show that 54% to 80% of the benchmarking duration (depending on the platform, and usage of the OpenSSL transport in the server, client, or both) is consumed in the Go runtime's cgocall() during invocations of functions in the underlying Open SSL library. In other words, the majority of the handshake duration in this implementation is consumed by C calls to the Open SSL library.

Appendix

Raw Results from the Raspberry Pi 4

Although benchmarking results vary from one run to another, representative benchmarking results for the Raspberry Pi 4 Model B - Ubuntu 20.10 - go1.15.5 arm64 platform were

Ran 80 of 80 Specs in 24.210 seconds
SUCCESS! -- 80 Passed | 0 Failed | 0 Pending | 0 Skipped
goos: linux
goarch: arm64
pkg: github.com/libp2p/go-libp2p-tls
BenchmarkHandshake_StdTLSServer_OpenSSLClient    	     724	   7609952 ns/op
BenchmarkHandshake_OpenSSLServer_StdTLSClient    	     843	   7052636 ns/op
BenchmarkHandshake_OpenSSLServer_OpenSSLClient   	     662	   9101175 ns/op
BenchmarkLatency_StdTLSServer_OpenSSLClient      	   41378	    145048 ns/op
BenchmarkLatency_OpenSSLServer_StdTLSClient      	   41408	    144747 ns/op
BenchmarkLatency_OpenSSLServer_OpenSSLClient     	   34060	    175600 ns/op
BenchmarkConnections_StdTLSServer_OpenSSLClient  	     597	  10112285 ns/op
BenchmarkConnections_OpenSSLServer_StdTLSClient  	     565	  10725068 ns/op
BenchmarkConnections_OpenSSLServer_OpenSSLClient 	     502	  11707676 ns/op
BenchmarkHandshake_StdTLSServer_StdTLSClient     	    1047	   5783148 ns/op
BenchmarkLatency_StdTLSServer_StdTLSClient       	   53679	    111170 ns/op
BenchmarkConnections_StdTLSServer_StdTLSClient   	     716	   8450207 ns/op
PASS
ok  	github.com/libp2p/go-libp2p-tls	120.103s

Raw Results from the Intel Core i5

Although benchmarking results vary from one run to another, representative benchmarking results for the Intel Core i5-10210U CPU @ 1.60 GHz - Ubuntu 20.04 - go1.15.5 amd64 platform were

Ran 80 of 80 Specs in 6.352 seconds
SUCCESS! -- 80 Passed | 0 Failed | 0 Pending | 0 Skipped
goos: linux
goarch: amd64
pkg: github.com/libp2p/go-libp2p-tls
BenchmarkHandshake_StdTLSServer_OpenSSLClient    	    4346	   1375445 ns/op
BenchmarkHandshake_OpenSSLServer_StdTLSClient    	    5589	   1164155 ns/op
BenchmarkHandshake_OpenSSLServer_OpenSSLClient   	    3192	   1853212 ns/op
BenchmarkLatency_StdTLSServer_OpenSSLClient      	  352849	     16116 ns/op
BenchmarkLatency_OpenSSLServer_StdTLSClient      	  356430	     16001 ns/op
BenchmarkLatency_OpenSSLServer_OpenSSLClient     	  286069	     20597 ns/op
BenchmarkConnections_StdTLSServer_OpenSSLClient  	    3710	   1594601 ns/op
BenchmarkConnections_OpenSSLServer_StdTLSClient  	    3499	   1712729 ns/op
BenchmarkConnections_OpenSSLServer_OpenSSLClient 	    2794	   2124285 ns/op
BenchmarkHandshake_StdTLSServer_StdTLSClient     	    8055	    729912 ns/op
BenchmarkLatency_StdTLSServer_StdTLSClient       	  609169	      9053 ns/op
BenchmarkConnections_StdTLSServer_StdTLSClient   	    5336	   1154001 ns/op
PASS
ok  	github.com/libp2p/go-libp2p-tls	89.455s