Benchmark and optimization of nerdctl images

[AkihiroSuda]

nerdctl images might be slower than before (which has been already slow).

We should do some benchmark and probably have --quick (--fast?, idk) flag to skip slow size calculation

Originally posted by @AkihiroSuda in #789 (comment)

[ningmingxiao]

I think nerdctl image -a or -o wide should show blobsize,at default it is not necessary to show it.

[fahedouch]

benchmark

Only Unpacked Size:

ntimes -n 10 _output/nerdctl images

REPOSITORY    TAG       IMAGE ID        CREATED           PLATFORM       SIZE         BLOB SIZE
alpine        latest    21a3deaa0d32    24 hours ago      linux/amd64    5.9 MiB      0.0 B
debian        latest    fb45fd4e25ab    14 minutes ago    linux/amd64    134.5 MiB    0.0 B
ubuntu        latest    669e010b58ba    13 minutes ago    linux/amd64    77.9 MiB     0.0 B

real average: 164.16197ms, max: 185.0302ms, min: 137.0472ms, std dev: 16.171173ms
real 99 percentile: 185.0302ms, 95 percentile: 185.0302ms, 50 percentile: 165.57285ms
user average: 54.0097ms, max: 64.725ms, min: 41.953ms, std dev: 7.904296ms
sys  average: 99.1182ms, max: 119.68ms, min: 79.722ms, std dev: 13.591908ms
flaky: 0%

Only Blob Size:

REPOSITORY    TAG       IMAGE ID        CREATED           PLATFORM       SIZE     BLOB SIZE
alpine        latest    21a3deaa0d32    24 hours ago      linux/amd64    0.0 B    2.7 MiB
debian        latest    fb45fd4e25ab    17 minutes ago    linux/amd64    0.0 B    52.4 MiB
ubuntu        latest    669e010b58ba    16 minutes ago    linux/amd64    0.0 B    27.2 MiB

real average: 157.88058ms, max: 236.3843ms, min: 142.0371ms, std dev: 28.001696ms
real 99 percentile: 236.3843ms, 95 percentile: 236.3843ms, 50 percentile: 149.58565ms
user average: 53.0145ms, max: 79.649ms, min: 43.332ms, std dev: 11.046308ms
sys  average: 86.4968ms, max: 100.921ms, min: 72.189ms, std dev: 8.907955ms
flaky: 0%

Both :

REPOSITORY    TAG       IMAGE ID        CREATED           PLATFORM       SIZE         BLOB SIZE
alpine        latest    21a3deaa0d32    24 hours ago      linux/amd64    5.9 MiB      2.7 MiB
debian        latest    fb45fd4e25ab    20 minutes ago    linux/amd64    134.5 MiB    52.4 MiB
ubuntu        latest    669e010b58ba    19 minutes ago    linux/amd64    77.9 MiB     27.2 MiB

real average: 184.36679ms, max: 217.7352ms, min: 170.1355ms, std dev: 15.003351ms
real 99 percentile: 217.7352ms, 95 percentile: 217.7352ms, 50 percentile: 174.8397ms
user average: 65.1658ms, max: 82.53ms, min: 52.764ms, std dev: 8.821194ms
sys  average: 106.5241ms, max: 112.316ms, min: 92.911ms, std dev: 5.815795ms
flaky: 0%

the difference between the 3 average is negligible I think It is useless to make a --quick flag

[AkihiroSuda]

Thanks, but we need to benchmark with more than 100 images, I guess

[fahedouch]

@AkihiroSuda I agree. But 100 is a huge number. Do you have a suggestion to get this number of image in one environnement 😅

[t1anz0ng]

I pulled a list of images from some of most pulled using docker hub API:

curl -s "https://hub.docker.com/v2/repositories/library/?page=1&page_size=100" | jq '.results | .[] | .name'

some of them are DEPRECATED (i.e SCRATCH). So in total, I have 96 images in my environment.

**Click to toggle full list of images**

REPOSITORY            TAG            IMAGE ID        CREATED           PLATFORM       SIZE          BLOB SIZE
jenkins/jenkins       latest         1f6e7ef75a54    19 minutes ago    linux/amd64    469.7 MiB     276.3 MiB
aerospike             latest         ffcb005eae9f    8 hours ago       linux/amd64    228.9 MiB     80.0 MiB
alpine                3.13           ccf92aa53bc6    8 days ago        linux/amd64    6.0 MiB       2.7 MiB
alpine                latest         7580ece7963b    8 hours ago       linux/amd64    5.9 MiB       2.7 MiB
arangodb              latest         ff91fb0eb65e    8 hours ago       linux/amd64    486.5 MiB     190.1 MiB
bonita                latest         1b28c73bbcde    34 minutes ago    linux/amd64    301.5 MiB     172.0 MiB
buildpack-deps        latest         4cbb6d56f192    9 hours ago       linux/amd64    858.2 MiB     307.5 MiB
busybox               latest         3614ca5eacf0    9 days ago        linux/amd64    1.3 MiB       759.3 KiB
cassandra             latest         d0ff1c79b5b3    8 hours ago       linux/amd64    347.6 MiB     144.0 MiB
celery                latest         5c236059192a    8 hours ago       linux/amd64    236.2 MiB     79.8 MiB
centos                latest         a27fd8080b51    9 hours ago       linux/amd64    244.4 MiB     79.7 MiB
cirros                latest         be6f5d1ab1e4    9 hours ago       linux/amd64    12.9 MiB      5.7 MiB
clojure               latest         3e1b7351b5e3    8 hours ago       linux/amd64    682.3 MiB     357.4 MiB
couchbase             latest         066a96ee394b    8 hours ago       linux/amd64    1.3 GiB       569.0 MiB
couchdb               latest         c4248560069d    30 minutes ago    linux/amd64    210.8 MiB     83.4 MiB
crate                 latest         3da3f761952d    8 hours ago       linux/amd64    844.1 MiB     353.0 MiB
crux                  latest         afcc4566ea59    9 hours ago       linux/amd64    467.9 MiB     152.2 MiB
debian                latest         2ce44bbc00a7    9 hours ago       linux/amd64    134.6 MiB     52.5 MiB
django                latest         5bfd3f442952    8 hours ago       linux/amd64    471.6 MiB     150.7 MiB
docker                latest         75290344f118    39 minutes ago    linux/amd64    281.6 MiB     89.8 MiB
drupal                latest         af84be385b9c    8 hours ago       linux/amd64    635.3 MiB     180.1 MiB
elixir                latest         e6d51468938b    28 minutes ago    linux/amd64    1.5 GiB       541.7 MiB
erlang                latest         e666e8cdb838    15 minutes ago    linux/amd64    1.5 GiB       536.9 MiB
fedora                latest         cbf627299e32    9 hours ago       linux/amd64    174.2 MiB     56.2 MiB
gazebo                latest         8f904171395e    8 hours ago       linux/amd64    2.0 GiB       581.9 MiB
gcc                   latest         ef7e8180a14e    8 hours ago       linux/amd64    1.3 GiB       439.4 MiB
ghost                 latest         90449eb5646c    8 hours ago       linux/amd64    648.9 MiB     140.4 MiB
glassfish             latest         9fb82b929fb6    8 hours ago       linux/amd64    0.0 B         334.4 MiB
golang                latest         9349ed889adb    8 hours ago       linux/amd64    1000.0 MiB    337.0 MiB
haproxy               latest         1e0bf13d9ee8    8 hours ago       linux/amd64    108.3 MiB     37.5 MiB
haskell               latest         7f83b45cc80d    8 hours ago       linux/amd64    2.7 GiB       650.7 MiB
hello-world           latest         53f1bbee2f52    8 hours ago       linux/amd64    20.0 KiB      6.9 KiB
hipache               latest         6d20ee9ddd59    9 hours ago       linux/amd64    0.0 B         146.8 MiB
httpd                 latest         75d370e19ec2    8 hours ago       linux/amd64    152.4 MiB     54.4 MiB
hylang                latest         0d3b6c28cb72    8 hours ago       linux/amd64    148.1 MiB     48.9 MiB
iojs                  latest         e9c867712191    8 hours ago       linux/amd64    683.2 MiB     242.3 MiB
irssi                 latest         3b638c10509b    8 hours ago       linux/amd64    186.7 MiB     49.0 MiB
java                  latest         c1ff613e8ba2    8 hours ago       linux/amd64    662.7 MiB     232.1 MiB
jetty                 latest         3e5d818cb467    8 hours ago       linux/amd64    475.0 MiB     244.6 MiB
joomla                latest         896a4b95d050    8 hours ago       linux/amd64    664.1 MiB     208.5 MiB
jruby                 latest         8157c7a110d5    8 hours ago       linux/amd64    308.1 MiB     116.5 MiB
julia                 latest         be7fa12413d4    8 hours ago       linux/amd64    520.0 MiB     158.5 MiB
kaazing-gateway       latest         2f308cf25e6d    8 hours ago       linux/amd64    302.3 MiB     128.8 MiB
lightstreamer         latest         99327f2f64e2    27 minutes ago    linux/amd64    722.5 MiB     367.0 MiB
mageia                latest         ee8deeb5ab22    8 hours ago       linux/amd64    324.4 MiB     105.4 MiB
mariadb               latest         88fcb7d92c7f    8 hours ago       linux/amd64    388.4 MiB     118.1 MiB
maven                 latest         d3b22b3d0904    8 hours ago       linux/amd64    534.6 MiB     257.6 MiB
memcached             latest         126afa4dc280    8 hours ago       linux/amd64    96.4 MiB      31.4 MiB
mongo                 latest         82302b063607    8 hours ago       linux/amd64    683.8 MiB     237.3 MiB
mono                  latest         705350895cae    8 hours ago       linux/amd64    788.2 MiB     242.6 MiB
mysql                 latest         152cf187a3ef    9 hours ago       linux/amd64    455.5 MiB     125.7 MiB
nats                  latest         4ee342fdd229    43 minutes ago    linux/amd64    11.4 MiB      4.4 MiB
neo4j                 latest         8ba4306cccb0    33 minutes ago    linux/amd64    544.2 MiB     343.6 MiB
neurodebian           latest         5bb987b78bd3    8 hours ago       linux/amd64    156.1 MiB     63.5 MiB
nginx                 1.19-alpine    07ab71a2c8e4    8 days ago        linux/amd64    25.7 MiB      9.4 MiB
nginx                 latest         1761fb5661e4    9 hours ago       linux/amd64    149.1 MiB     54.1 MiB
node                  latest         c9504e6bdd04    9 hours ago       linux/amd64    1.0 GiB       353.3 MiB
nuxeo                 latest         28de083ab4dc    31 minutes ago    linux/amd64    1.8 GiB       911.8 MiB
odoo                  latest         485464c5a716    8 hours ago       linux/amd64    1.6 GiB       530.4 MiB
orientdb              latest         78fe0f955ad8    26 minutes ago    linux/amd64    395.6 MiB     203.9 MiB
owncloud              latest         173811cb4c40    8 hours ago       linux/amd64    655.7 MiB     210.9 MiB
percona               latest         8e77cd4bdbed    8 hours ago       linux/amd64    662.8 MiB     235.2 MiB
perl                  latest         093c0d91185d    8 hours ago       linux/amd64    920.8 MiB     322.0 MiB
photon                latest         1db94cc46388    30 minutes ago    linux/amd64    36.8 MiB      15.3 MiB
php-zendserver        latest         487845576d84    8 hours ago       linux/amd64    1.0 GiB       375.3 MiB
php                   latest         fd48f3c38643    8 hours ago       linux/amd64    488.3 MiB     161.2 MiB
piwik                 latest         f7b6e52f7934    29 minutes ago    linux/amd64    481.2 MiB     159.2 MiB
postgres              latest         3e2eba0a6efb    9 hours ago       linux/amd64    383.4 MiB     131.4 MiB
pypy                  latest         cf7c9b5a5c17    8 hours ago       linux/amd64    995.3 MiB     342.7 MiB
python                latest         ce21f64c4c3a    8 hours ago       linux/amd64    946.2 MiB     335.0 MiB
r-base                latest         3cd83a271bac    8 hours ago       linux/amd64    793.8 MiB     324.6 MiB
rabbitmq              latest         68cf8d3329e0    8 hours ago       linux/amd64    237.4 MiB     95.2 MiB
rails                 latest         aec52fe81ff0    8 hours ago       linux/amd64    878.9 MiB     303.1 MiB
rakudo-star           latest         4812508caf81    8 hours ago       linux/amd64    477.3 MiB     154.4 MiB
redis                 alpine3.13     6833ca04ec87    2 weeks ago       linux/amd64    33.8 MiB      10.4 MiB
redis                 latest         ed8cba11c094    9 hours ago       linux/amd64    122.7 MiB     40.4 MiB
redmine               latest         eff62a11e469    8 hours ago       linux/amd64    637.9 MiB     226.1 MiB
registry              latest         c631a581c615    9 hours ago       linux/amd64    24.5 MiB      8.8 MiB
rethinkdb             latest         3525b9b8277f    8 hours ago       linux/amd64    129.0 MiB     45.7 MiB
rocket.chat           latest         532082b9156c    41 minutes ago    linux/amd64    1.3 GiB       262.8 MiB
ros                   latest         89a1c3b529c4    8 hours ago       linux/amd64    815.9 MiB     250.5 MiB
ruby                  latest         db317226be7a    8 hours ago       linux/amd64    921.4 MiB     338.4 MiB
sentry                latest         5a9fb82278c8    8 hours ago       linux/amd64    912.9 MiB     251.6 MiB
solr                  latest         08fff33aefac    34 minutes ago    linux/amd64    534.3 MiB     308.7 MiB
sonarqube             latest         b89044d191da    8 hours ago       linux/amd64    513.4 MiB     349.5 MiB
sourcemage            latest         f39c8eeadfc8    35 minutes ago    linux/amd64    938.4 MiB     251.0 MiB
swarm                 latest         2de8883e2933    8 hours ago       linux/amd64    12.2 MiB      3.7 MiB
thrift                latest         c05261d526cd    8 hours ago       linux/amd64    126.9 MiB     41.7 MiB
tomcat                latest         9edc5c51c2de    8 hours ago       linux/amd64    479.1 MiB     243.7 MiB
tomee                 latest         90b08fdfab43    38 minutes ago    linux/amd64    367.8 MiB     162.9 MiB
ubuntu-debootstrap    latest         e74053a4261a    8 hours ago       linux/amd64    101.2 MiB     33.3 MiB
ubuntu-upstart        latest         780f166fa5f9    9 hours ago       linux/amd64    282.4 MiB     97.0 MiB
ubuntu                latest         b6b83d3c3317    9 hours ago       linux/amd64    83.4 MiB      29.0 MiB
websphere-liberty     latest         6ec97f5e5029    8 hours ago       linux/amd64    701.8 MiB     441.0 MiB
wordpress             latest         999392cfea3c    9 hours ago       linux/amd64    624.8 MiB     204.7 MiB

I did @fahedouch experiment:
cc: @AkihiroSuda

BOTH

REPOSITORY            TAG            IMAGE ID        CREATED           PLATFORM       SIZE          BLOB SIZE
ubuntu-debootstrap    latest         e74053a4261a    8 hours ago       linux/amd64    101.2 MiB     33.3 MiB
......
wordpress             latest         999392cfea3c    8 hours ago       linux/amd64    624.8 MiB     204.7 MiB

real average: 3.26550947s, max: 3.293986785s, min: 3.239568373s, std dev: 17.687189ms
real 99 percentile: 3.293986785s, 95 percentile: 3.293986785s, 50 percentile: 3.26016171s
user average: 2.3691691s, max: 2.431819s, min: 2.318302s, std dev: 38.575007ms
sys  average: 1.3688926s, max: 1.444045s, min: 1.293038s, std dev: 49.071786ms

Without BlobSize( comment image.Size(ctx))

REPOSITORY            TAG            IMAGE ID        CREATED           PLATFORM       SIZE          BLOB SIZE
java                  latest         c1ff613e8ba2    8 hours ago       linux/amd64    662.7 MiB     0.0 B
...
wordpress             latest         999392cfea3c    8 hours ago       linux/amd64    624.8 MiB     0.0 B

real average: 2.986185244s, max: 3.01026741s, min: 2.964963446s, std dev: 12.432606ms
real 99 percentile: 3.01026741s, 95 percentile: 3.01026741s, 50 percentile: 2.983283399s
user average: 2.1341679s, max: 2.221924s, min: 2.033311s, std dev: 56.500939ms
sys  average: 1.2875114s, max: 1.377556s, min: 1.228951s, std dev: 44.839664ms

Without Unpacked Size( comment unpackedImageSize(ctx, x.snapshotter, image))

REPOSITORY            TAG            IMAGE ID        CREATED           PLATFORM       SIZE          BLOB SIZE
jenkins/jenkins       latest         1f6e7ef75a54    18 minutes ago    linux/amd64    0.0 B    276.3 MiB
...
wordpress             latest         999392cfea3c    8 hours ago       linux/amd64    0.0 B    204.7 MiB

real average: 2.079441195s, max: 2.100727932s, min: 1.989394178s, std dev: 32.790733ms
real 99 percentile: 2.100727932s, 95 percentile: 2.100727932s, 50 percentile: 2.083640085s
user average: 1.5221273s, max: 1.624942s, min: 1.371136s, std dev: 68.459596ms
sys  average: 863.0735ms, max: 908.323ms, min: 797.141ms, std dev: 38.225147ms

Without both ( comment both)

REPOSITORY            TAG            IMAGE ID        CREATED           PLATFORM       SIZE     BLOB SIZE
neo4j                 latest         8ba4306cccb0    44 minutes ago    linux/amd64    0.0 B    0.0 B
...
percona               latest         8e77cd4bdbed    8 hours ago       linux/amd64    0.0 B    0.0 B

real average: 1.808876959s, max: 1.837062398s, min: 1.793315846s, std dev: 15.067907ms
real 99 percentile: 1.837062398s, 95 percentile: 1.837062398s, 50 percentile: 1.80513215s
user average: 1.3167565s, max: 1.369457s, min: 1.242427s, std dev: 44.493909ms
sys  average: 749.8885ms, max: 855.136ms, min: 661.094ms, std dev: 62.384582ms

---

seems unpackedImageSize cost a lot of time?
About 44% faster if we skip size calculation.

nerdctl version >= 0.22.0

[apostasie]

Reviving this issue, and sharing notes here:

I was working on adding caching to the snapshotter for commands like images (so that we do not make several Stat and Usage calls for the same objects), and started looking into image list overall.

Our current implementation of image list first calls on images.List.

Then, for each image, we call image.Platforms(), which:

• reads the manifest blob (again)
• reads the blob of every children of the target that are relevant - if the image is a straight manifest, it will read the config

Then, for each image and each architecture, we call on images.Check, which:

• reads the manifest blob (again)
• possibly reads the config blob as well
• then ReaderAt on every "required" resource (eg: any linked manifest)

ReaderAt does:

• os.Stat
• os.Open

Then we call containerd.NewImageWithPlatform, then Config() on the resulting image, which will read the config blob again.

Then we call RootFS for the size calculation, which again will read the config blob (twice).

Then we call image.Size for the blob size calculation, which will again read the blob of every children of the target.

That is a lot of duplicate filesystem calls.

A quick auditd test confirms the above and reports about 60 filesystem hits for Alpine here (alpine contains one layer and 8 architectures).
Whether or not this is entirely accurate, or my reading of the code is 100% right, this is still a lot of filesystem access for one single image.

Suggestion:

• we could bypass entirely all the containerd helpers methods and use the content store directly to avoid all of that - containerd helpers are great for single objects where io does not matter much, but just not good for the task at hand and clearly problematic at scale
• we change the logic so that we stop repeating these operations on the same image for every platform, and read things once wherever possible

That should bring down the number of filesystem access significantly.

Here are some highly not scientific numbers against main and against my current work in progress:

For 500 identical, single platform images (tagged from apostasie/test), time will give:

• currently:
  • ~1.5 seconds
• with rewrite:
  • ~0.4 seconds

For 500 identical, multi platform images (all tagged from alpine:latest, just the current platform):

• currently:
  • ~3.3 seconds
• with rewrite:
  • ~0.8 seconds

For 500 identical, multi truly platform images (all tagged from alpine:latest, --all-platform):

• currently:
  • ~12 seconds
• with rewrite:
  • ~2.9 seconds

Using the image list from above, with code instrumentation, printImage execution time gets divided by 3 (from about 1 second to 0.3 second), although time only shows about 20% improvement overall.

The PR is not ready to be merged and more a proof of concept, but would like to know what you folks think.

Cheers.

[apostasie]

On today's main (873a087), with the list of images provided above (84 images currently), on a lima setup / M1 Max:

real average: 408.938725ms, max: 430.797757ms, min: 391.571417ms, std dev: 14.795711ms
real 99 percentile: 430.797757ms, 95 percentile: 430.797757ms, 50 percentile: 398.672759ms
user average: 167.9466ms, max: 201.888ms, min: 133.506ms, std dev: 25.245575ms
sys  average: 151.5746ms, max: 173.962ms, min: 110.628ms, std dev: 22.000527ms
flaky: 0%