Skip to content

Latest commit

 

History

History

bench

README.md

Sample Benchmark Results

This test simply measures the performance of several reverse proxy through HTTP/1.1 by the following command using rewrk.

$ rewrk -c 512 -t 4 -d 15s -h http://localhost:8080 --pct

Tests on linux/arm64/v8

Done at Jul. 15, 2023

Environment

  • rpxy commit id: 1da7e5bfb77d1ce4ee8d6cfc59b1c725556fc192
  • Docker Desktop 4.21.1 (114176)
  • ReWrk 0.3.2
  • Macbook Pro '14 (2021, M1 Max, 64GB RAM)

The docker images of nginx and caddy for linux/arm64/v8 are pulled from the official registry.

Result for rpxy, nginx and caddy

----------------------------
Benchmark on rpxy
Beginning round 1...
Benchmarking 512 connections @ http://localhost:8080 for 15 second(s)
  Latencies:
    Avg      Stdev    Min      Max
    19.64ms  8.85ms   0.67ms   113.22ms
  Requests:
    Total: 390078  Req/Sec: 26011.25
  Transfer:
    Total: 304.85 MB Transfer Rate: 20.33 MB/Sec
+ --------------- + --------------- +
|   Percentile    |   Avg Latency   |
+ --------------- + --------------- +
|      99.9%      |     79.24ms     |
|       99%       |     54.28ms     |
|       95%       |     42.50ms     |
|       90%       |     37.82ms     |
|       75%       |     31.54ms     |
|       50%       |     26.37ms     |
+ --------------- + --------------- +

721 Errors: error shutting down connection: Socket is not connected (os error 57)

sleep 3 secs
----------------------------
Benchmark on nginx
Beginning round 1...
Benchmarking 512 connections @ http://localhost:8090 for 15 second(s)
  Latencies:
    Avg      Stdev    Min      Max
    33.26ms  15.18ms  1.40ms   118.94ms
  Requests:
    Total: 230268  Req/Sec: 15356.08
  Transfer:
    Total: 186.77 MB Transfer Rate: 12.46 MB/Sec
+ --------------- + --------------- +
|   Percentile    |   Avg Latency   |
+ --------------- + --------------- +
|      99.9%      |     99.91ms     |
|       99%       |     83.74ms     |
|       95%       |     70.67ms     |
|       90%       |     64.03ms     |
|       75%       |     54.32ms     |
|       50%       |     45.19ms     |
+ --------------- + --------------- +

677 Errors: error shutting down connection: Socket is not connected (os error 57)

sleep 3 secs
----------------------------
Benchmark on caddy
Beginning round 1...
Benchmarking 512 connections @ http://localhost:8100 for 15 second(s)
  Latencies:
    Avg      Stdev    Min      Max
    48.51ms  50.74ms  0.34ms   554.58ms
  Requests:
    Total: 157239  Req/Sec: 10485.98
  Transfer:
    Total: 125.99 MB Transfer Rate: 8.40 MB/Sec
+ --------------- + --------------- +
|   Percentile    |   Avg Latency   |
+ --------------- + --------------- +
|      99.9%      |    473.82ms     |
|       99%       |    307.16ms     |
|       95%       |    212.28ms     |
|       90%       |    169.05ms     |
|       75%       |    115.92ms     |
|       50%       |     80.24ms     |
+ --------------- + --------------- +

708 Errors: error shutting down connection: Socket is not connected (os error 57)

Results on linux/amd64

Done at Jul. 24, 2023

Environment

  • rpxy commit id: 7c0945a5124418aa9a1024568c1989bb77cf312f
  • Docker Desktop 4.21.1 (114176)
  • ReWrk 0.3.2 and Wrk 0.4.2
  • iMac '27 (2020, 10-Core Intel Core i9, 128GB RAM)

The docker images of nginx and caddy for linux/amd64 were pulled from the official registry. For Sozu, the official docker image from its developers was still version 0.11.0 (currently the latest version is 0.15.2). So we built it by ourselves locally using the Sozu's official Dockerfile.

Also, when Sozu is configured as an HTTP reverse proxy, it cannot handle HTTP request messages emit from ReWrk due to hostname parsing errors though it can correctly handle messages dispatched from curl and browsers. So, we additionally test using Wrk to examine Sozu with the following command.

$ wrk -c 512 -t 4 -d 15s http://localhost:8110

Result

With ReWrk for rpxy, nginx and caddy

----------------------------
Benchmark [x86_64] with ReWrk
----------------------------
Benchmark on rpxy
Beginning round 1...
Benchmarking 512 connections @ http://localhost:8080 for 15 second(s)
  Latencies:
    Avg      Stdev    Min      Max
    20.37ms  8.95ms   1.63ms   160.27ms
  Requests:
    Total: 376345  Req/Sec: 25095.19
  Transfer:
    Total: 295.61 MB Transfer Rate: 19.71 MB/Sec
+ --------------- + --------------- +
|   Percentile    |   Avg Latency   |
+ --------------- + --------------- +
|      99.9%      |    112.50ms     |
|       99%       |     61.33ms     |
|       95%       |     44.26ms     |
|       90%       |     38.74ms     |
|       75%       |     32.00ms     |
|       50%       |     26.82ms     |
+ --------------- + --------------- +

626 Errors: error shutting down connection: Socket is not connected (os error 57)

sleep 3 secs
----------------------------
Benchmark on nginx
Beginning round 1...
Benchmarking 512 connections @ http://localhost:8090 for 15 second(s)
  Latencies:
    Avg      Stdev    Min      Max
    23.45ms  12.42ms  1.18ms   154.44ms
  Requests:
    Total: 326685  Req/Sec: 21784.73
  Transfer:
    Total: 265.22 MB Transfer Rate: 17.69 MB/Sec
+ --------------- + --------------- +
|   Percentile    |   Avg Latency   |
+ --------------- + --------------- +
|      99.9%      |     96.85ms     |
|       99%       |     73.93ms     |
|       95%       |     57.57ms     |
|       90%       |     50.36ms     |
|       75%       |     40.57ms     |
|       50%       |     32.70ms     |
+ --------------- + --------------- +

657 Errors: error shutting down connection: Socket is not connected (os error 57)

sleep 3 secs
----------------------------
Benchmark on caddy
Beginning round 1...
Benchmarking 512 connections @ http://localhost:8100 for 15 second(s)
  Latencies:
    Avg      Stdev    Min      Max
    45.71ms  50.47ms  0.88ms   908.49ms
  Requests:
    Total: 166917  Req/Sec: 11129.80
  Transfer:
    Total: 133.77 MB Transfer Rate: 8.92 MB/Sec
+ --------------- + --------------- +
|   Percentile    |   Avg Latency   |
+ --------------- + --------------- +
|      99.9%      |    608.92ms     |
|       99%       |    351.18ms     |
|       95%       |    210.56ms     |
|       90%       |    162.68ms     |
|       75%       |    106.97ms     |
|       50%       |     73.90ms     |
+ --------------- + --------------- +

646 Errors: error shutting down connection: Socket is not connected (os error 57)

sleep 3 secs

With Wrk for rpxy, nginx, caddy and sozu

----------------------------
Benchmark [x86_64] with Wrk
----------------------------
Benchmark on rpxy
Running 15s test @ http://localhost:8080
  4 threads and 512 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    18.68ms    8.09ms 122.64ms   74.03%
    Req/Sec     6.95k   815.23     8.45k    83.83%
  414819 requests in 15.01s, 326.37MB read
  Socket errors: connect 0, read 608, write 0, timeout 0
Requests/sec:  27627.79
Transfer/sec:     21.74MB

sleep 3 secs
----------------------------
Benchmark on nginx
Running 15s test @ http://localhost:8090
  4 threads and 512 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    23.34ms   13.80ms 126.06ms   74.66%
    Req/Sec     5.71k   607.41     7.07k    73.17%
  341127 requests in 15.03s, 277.50MB read
  Socket errors: connect 0, read 641, write 0, timeout 0
Requests/sec:  22701.54
Transfer/sec:     18.47MB

sleep 3 secs
----------------------------
Benchmark on caddy
Running 15s test @ http://localhost:8100
  4 threads and 512 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    54.19ms   55.63ms 674.53ms   88.55%
    Req/Sec     2.92k     1.40k    5.57k    56.17%
  174748 requests in 15.03s, 140.61MB read
  Socket errors: connect 0, read 660, write 0, timeout 0
  Non-2xx or 3xx responses: 70
Requests/sec:  11624.63
Transfer/sec:      9.35MB

sleep 3 secs
----------------------------
Benchmark on sozu
Running 15s test @ http://localhost:8110
  4 threads and 512 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    19.78ms    4.89ms  98.09ms   76.88%
    Req/Sec     6.49k   824.75     8.11k    76.17%
  387744 requests in 15.02s, 329.11MB read
  Socket errors: connect 0, read 647, write 0, timeout 0
Requests/sec:  25821.93
Transfer/sec:     21.92MB