benchmark: docker run, nerdctl run, podman run

[AkihiroSuda]

Not an issue, just sharing rough benchmark result.

TL; DR: nerdctl run is slower than docker run by about 60ms on average. Negligible in most cases.

Rootful

Docker > nerdctl > Podman

# ntimes -n 10 docker run --rm --runtime=runc alpine true

real average: 556.274676ms, max: 646.350244ms, min: 513.601646ms, std dev: 39.858611ms
real 99 percentile: 646.350244ms, 95 percentile: 646.350244ms, 50 percentile: 539.802737ms
user average: 46.4394ms, max: 63.403ms, min: 29.504ms, std dev: 9.953209ms
sys  average: 23.3963ms, max: 36.935ms, min: 14.035ms, std dev: 7.870294ms
flaky: 0%

# ntimes -n 10 nerdctl run --rm --runtime=runc alpine true

real average: 618.869742ms, max: 659.882839ms, min: 587.541024ms, std dev: 20.434296ms
real 99 percentile: 659.882839ms, 95 percentile: 659.882839ms, 50 percentile: 616.619364ms
user average: 87.5675ms, max: 114.714ms, min: 63.692ms, std dev: 14.569138ms
sys  average: 47.6504ms, max: 68.963ms, min: 31.164ms, std dev: 13.37964ms
flaky: 0%

# ntimes -n 10 podman run --rm --runtime=runc alpine true

real average: 993.334419ms, max: 2.071687601s, min: 752.553124ms, std dev: 441.94414ms
real 99 percentile: 2.071687601s, 95 percentile: 2.071687601s, 50 percentile: 765.565238ms
user average: 592.8803ms, max: 1.220949s, min: 371.482ms, std dev: 301.697256ms
sys  average: 226.6761ms, max: 316.794ms, min: 177.988ms, std dev: 44.979952ms
flaky: 0%

Rootless

Podman > Docker > nerdctl

Not sure why rootless Podman is much faster than rootful Podman, though.

$ ntimes -n 10 docker run --rm --runtime=runc alpine true

real average: 569.11061ms, max: 609.74447ms, min: 540.242145ms, std dev: 23.482413ms
real 99 percentile: 609.74447ms, 95 percentile: 609.74447ms, 50 percentile: 561.673241ms
user average: 44.9384ms, max: 61.429ms, min: 22.366ms, std dev: 12.853759ms
sys  average: 20.5545ms, max: 40.26ms, min: 7.222ms, std dev: 10.25749ms
flaky: 0%

$ ntimes -n 10 nerdctl run --rm --runtime=runc alpine true

real average: 651.934524ms, max: 694.843773ms, min: 626.688207ms, std dev: 22.905614ms
real 99 percentile: 694.843773ms, 95 percentile: 694.843773ms, 50 percentile: 648.765191ms
user average: 105.8108ms, max: 138.74ms, min: 76.018ms, std dev: 19.734353ms
sys  average: 60.5209ms, max: 72.859ms, min: 45.349ms, std dev: 9.563961ms
flaky: 0%


$ ntimes -n 10 podman run --rm --runtime=runc alpine true

real average: 406.569319ms, max: 473.388183ms, min: 354.530393ms, std dev: 55.747798ms
real 99 percentile: 473.388183ms, 95 percentile: 473.388183ms, 50 percentile: 365.584304ms
user average: 158.0039ms, max: 181.352ms, min: 132.891ms, std dev: 15.041919ms
sys  average: 75.095ms, max: 103.707ms, min: 50.294ms, std dev: 16.179126ms
flaky: 0%

---

Version:

• Docker: moby/moby@459d0dfbbb
• nerdctl: b235a09
• Podman: v4.0.0-rc3
• containerd: v1.6.0-rc.1
• runc: v1.1.0
• Host: Ubuntu 21.10, kernel 5.13 (so, fuse-overlayfs is not used for rootless)

[junnplus]

$ hyperfine --runs 20 'sudo nerdctl run --rm alpine true'
Benchmark 1: sudo nerdctl run --rm alpine true
  Time (mean ± σ):     338.8 ms ±  33.4 ms    [User: 2.0 ms, System: 2.7 ms]
  Range (min … max):   304.8 ms … 458.2 ms    20 runs

$ hyperfine --runs 20 'sudo docker run --rm alpine true'
Benchmark 1: sudo docker run --rm alpine true
  Time (mean ± σ):     482.6 ms ±  28.1 ms    [User: 1.4 ms, System: 3.4 ms]
  Range (min … max):   429.5 ms … 539.1 ms    20 runs

Version:

• Docker: 20.10.17
• nerdctl: v0.22.0
• containerd: v1.6.4
• runc: 1.1.2
• Host: ubuntu 22.04, kernel 5.15