From 3a03f82c205ac0766e2c2ac65c388a04fd62e015 Mon Sep 17 00:00:00 2001 From: Marco Munizaga Date: Thu, 10 Nov 2022 12:38:05 -0800 Subject: [PATCH 1/6] Add perf spec --- perf/perf.md | 121 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 121 insertions(+) create mode 100644 perf/perf.md diff --git a/perf/perf.md b/perf/perf.md new file mode 100644 index 000000000..198a4618e --- /dev/null +++ b/perf/perf.md @@ -0,0 +1,121 @@ +# Perf + +| Lifecycle Stage | Maturity | Status | Latest Revision | +| --------------- | ------------- | ------ | --------------- | +| 1A | Working Draft | Active | r0, 2022-11-16 | + +Authors: [@marcopolo] + +Interest Group: [@marcopolo], [@mxinden], [@marten-seemann] + +[@marcopolo]: https://github.com/mxinden +[@mxinden]: https://github.com/mxinden +[@marten-seemann]: https://github.com/marten-seemann + +# Table of Contents +- [Context](#context) +- [Protocol](#protocol) +- [Benchmarks](#benchmarks) + - [Single connection throughput](#single-connection-throughput) + - [Handshakes per second](#handshakes-per-second) +- [Security Considerations](#security-considerations) +- [Prior Art](#prior-art) + +# Context + +The `perf` protocol represents a standard benchmarking protocol that we can use +to talk about performance within and across libp2p implementations. This lets us +analyze peformance, guide improvements, and protect against regressions. + +# Protocol + +The `/perf/1.0.0` protocol (from here on referred to as simply `perf`) is a +client driven set of benchmarks. To not reinvent the wheel, this perf protocol +is almost identical to Nick Bank's [_QUIC Performance_ +Internet-Draft](https://datatracker.ietf.org/doc/html/draft-banks-quic-performance#section-2.3) +but adapted to libp2p. + + +The protocol is as a follows: + +Client: + +1. Open a libp2p stream to the server. +2. Tell the server how many bytes we want the server to send us as a single + big-endian uint64 number. Zero is a valid number, so is the max uint64 value. +3. Write some amount of data to the stream. +4. Close the write side of our stream. +5. Read from the read side of the stream. This + should be the same number of bytes as we told the server in step 2. + +Server, on handling a new `perf` stream: +1. Read the big-endian uint64 number. This is how many bytes we'll send back. +2. Read from the stream until we get an `EOF` (client's write side was closed). +3. Send the number of bytes defined in step 1 back to the client. +4. Close the stream. + +# Benchmarks + +The above protocol is flexible enough to run the following benchmarks and more. +The exact specifics of the benchmark (e.g. how much data to download or for how +long) are left up to the benchmark implementation. Consider these rough +guidelines for how to run one of these benchmarks. + +Other benchmarks can be run with the same protocol. The following benchmarks +have immediate usefulness, but other benchmarks can be added as we find them +useful. Consult the [_QUIC Performance_ +Internet-Draft](https://datatracker.ietf.org/doc/html/draft-banks-quic-performance#section-2.3) +for some other benchmarks (called _scenarios_ in the document). + +## Single connection throughput + +For an upload test, the client sets the the server response size 0 bytes, writes +some amount of data and closes the stream. + +For a download test, the client sets the server response size to N bytes, and +closes the write side of the data. + +The measurements are gathered and reported by the client by measuring how many +bytes were transferred by the total time it took from stream open to stream +close. + +A timer based variant is also possible where we see how much data a client can +upload or download within a specific time. For upload it's the same as before +and the client closes the stream after the timer ends. For download, the client +should request a response size of max uint64, then close the stream after the +timer ends. + +## Handshakes per second + +This benchmark measures connection setup efficiency. A transport that takes many +RTTs will perform worse here than one that takes fewer. + +To run this benchmark: +1. Set up N clients +2. Each client opens K connections/s to a single server +3. once a connection is established, the client closes it and establishes + another one. + +handshakes per second are calculated by taking the total number of connections +successfully established and divide it by the time period of the test. + +# Security Considerations + +Since this protocol lets clients ask servers to do significant work, it +SHOULD NOT be enabled by default in any implementation. Users are advised not to +enable this on publicly reachable nodes. + +Authentacting by Peer ID could mitigate the security concern by only allowing +trusted clients to use the protocol. Support for this is left to the implementation. + +# Prior Art + +As mentioned above, this document is inspired by Nick Bank's: [QUIC Performance Internet-Draft](https://datatracker.ietf.org/doc/html/draft-banks-quic-performance) + +[iperf](https://iperf.fr) + +[@mxinden's libp2p perf](https://github.com/mxinden/libp2p-perf) + +[@marten-seemann's libp2p perf test](https://github.com/marten-seemann/libp2p-perf-test/) + +[@vyzo's libp2p perf test](https://github.com/vyzo/libp2p-perf-test/) From c52e047e2b1dc71de83c71ec939d23252a4ecfbb Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Thu, 2 Mar 2023 14:48:24 -0800 Subject: [PATCH 2/6] perf: clarify that upload and download are run sequentially --- perf/perf.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/perf/perf.md b/perf/perf.md index 198a4618e..c48c2f21e 100644 --- a/perf/perf.md +++ b/perf/perf.md @@ -34,6 +34,8 @@ client driven set of benchmarks. To not reinvent the wheel, this perf protocol is almost identical to Nick Bank's [_QUIC Performance_ Internet-Draft](https://datatracker.ietf.org/doc/html/draft-banks-quic-performance#section-2.3) but adapted to libp2p. +The protocol first performs an upload of a client-chosen amount of bytes. Once +that upload has finished, the server sends back as many bytes as the client requested. The protocol is as a follows: @@ -43,16 +45,17 @@ Client: 1. Open a libp2p stream to the server. 2. Tell the server how many bytes we want the server to send us as a single big-endian uint64 number. Zero is a valid number, so is the max uint64 value. -3. Write some amount of data to the stream. +3. Optional: Write some amount of data to the stream. 4. Close the write side of our stream. 5. Read from the read side of the stream. This should be the same number of bytes as we told the server in step 2. Server, on handling a new `perf` stream: -1. Read the big-endian uint64 number. This is how many bytes we'll send back. +1. Read the big-endian uint64 number. This is how many bytes we'll send back in step 3. 2. Read from the stream until we get an `EOF` (client's write side was closed). -3. Send the number of bytes defined in step 1 back to the client. -4. Close the stream. +3. Send the number of bytes defined in step 1 back to the client. This MUST NOT be run + concurrently with step 2. +5. Close the stream. # Benchmarks From d137441ff2067a4f5da8b1100855fad0a355a383 Mon Sep 17 00:00:00 2001 From: Marco Munizaga Date: Wed, 31 May 2023 09:47:17 -0700 Subject: [PATCH 3/6] Update perf/perf.md Co-authored-by: Thomas Eizinger --- perf/perf.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/perf/perf.md b/perf/perf.md index c48c2f21e..187edfd07 100644 --- a/perf/perf.md +++ b/perf/perf.md @@ -45,7 +45,8 @@ Client: 1. Open a libp2p stream to the server. 2. Tell the server how many bytes we want the server to send us as a single big-endian uint64 number. Zero is a valid number, so is the max uint64 value. -3. Optional: Write some amount of data to the stream. +3. Write some amount of data to the stream. + Zero is a valid amount. 4. Close the write side of our stream. 5. Read from the read side of the stream. This should be the same number of bytes as we told the server in step 2. From 21ee92ff65e9a1f72e05050e2be3e73c6d0eb08b Mon Sep 17 00:00:00 2001 From: Marco Munizaga Date: Wed, 31 May 2023 09:50:10 -0700 Subject: [PATCH 4/6] Typos --- perf/perf.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/perf/perf.md b/perf/perf.md index 187edfd07..26777895e 100644 --- a/perf/perf.md +++ b/perf/perf.md @@ -8,7 +8,7 @@ Authors: [@marcopolo] Interest Group: [@marcopolo], [@mxinden], [@marten-seemann] -[@marcopolo]: https://github.com/mxinden +[@marcopolo]: https://github.com/marcopolo [@mxinden]: https://github.com/mxinden [@marten-seemann]: https://github.com/marten-seemann @@ -73,7 +73,7 @@ for some other benchmarks (called _scenarios_ in the document). ## Single connection throughput -For an upload test, the client sets the the server response size 0 bytes, writes +For an upload test, the client sets the the server response size to 0 bytes, writes some amount of data and closes the stream. For a download test, the client sets the server response size to N bytes, and From b396c253ca89f3d80e9ca1f3001c09fd99cadd0e Mon Sep 17 00:00:00 2001 From: Marco Munizaga Date: Wed, 31 May 2023 09:52:20 -0700 Subject: [PATCH 5/6] Add info about what the bytes are --- perf/perf.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/perf/perf.md b/perf/perf.md index 26777895e..53a76348d 100644 --- a/perf/perf.md +++ b/perf/perf.md @@ -35,7 +35,13 @@ is almost identical to Nick Bank's [_QUIC Performance_ Internet-Draft](https://datatracker.ietf.org/doc/html/draft-banks-quic-performance#section-2.3) but adapted to libp2p. The protocol first performs an upload of a client-chosen amount of bytes. Once -that upload has finished, the server sends back as many bytes as the client requested. +that upload has finished, the server sends back as many bytes as the client +requested. + +The bytes themselves should be a predetermined arbitrary set of bytes. Zero is +fine, but so is random bytes (as long as it's not a different set of random +bytes, because then you may be limited by how fast you can generate random +bytes). The protocol is as a follows: From 4f9ae25ca90816805aefa7f729fdcf4439c3c0bc Mon Sep 17 00:00:00 2001 From: Marco Munizaga Date: Wed, 31 May 2023 10:02:37 -0700 Subject: [PATCH 6/6] Typo --- perf/perf.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/perf/perf.md b/perf/perf.md index 53a76348d..061db76c6 100644 --- a/perf/perf.md +++ b/perf/perf.md @@ -106,7 +106,7 @@ To run this benchmark: 3. once a connection is established, the client closes it and establishes another one. -handshakes per second are calculated by taking the total number of connections +Handshakes per second are calculated by taking the total number of connections successfully established and divide it by the time period of the test. # Security Considerations