Raw SSD performance benchmark #9147

pugachAG · 2023-06-06T08:21:05Z

pugachAG
Jun 6, 2023
Collaborator

The goal is to analyse raw SSD performance on different platforms. This is important when reasoning and benchmarking neard storage.

In the context of storage we are interested in random read latency since that is the access pattern behind rocksdb get operations for State column. Also we are interested in the max number of parallel reads that can be done without compromising latency, which is important for background trie reads to work.

Related work: Low level storage benchmark results.

Setup

fio tool is used to execute random reads workload.

Data is prepared with the following config:

[write_job]
filename=/home/ubuntu/.near/data/fio_test
size=450Gb

size is adjusted depending on the SSD size to make it as close to being full as possible.

The following config is used to execute the benchmark:

[read_job]
ioengine=psync
filename=/home/ubuntu/.near/data/fio_test
rw=randread
bs=4k
direct=1
group_reporting
time_based
runtime=60
numjobs=8

GCP persistant SSD

Persistent SSD is currently the default solution for rocksdb data (~/.near/data).
The latency seems to be pretty unstable and I got different results for when running the benchmark on different days.

Result 1

read_job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.16
Starting 8 processes
Jobs: 8 (f=8): [r(8)][100.0%][r=103MiB/s][r=26.3k IOPS][eta 00m:00s]
read_job: (groupid=0, jobs=8): err= 0: pid=196386: Sun Jun  4 14:50:15 2023
  read: IOPS=26.6k, BW=104MiB/s (109MB/s)(6234MiB/60003msec)
    clat (usec): min=120, max=6698, avg=299.31, stdev=128.04
     lat (usec): min=121, max=6698, avg=299.42, stdev=128.04
    clat percentiles (usec):
     |  1.00th=[  167],  5.00th=[  186], 10.00th=[  200], 20.00th=[  219],
     | 30.00th=[  235], 40.00th=[  251], 50.00th=[  273], 60.00th=[  293],
     | 70.00th=[  322], 80.00th=[  359], 90.00th=[  424], 95.00th=[  490],
     | 99.00th=[  717], 99.50th=[  938], 99.90th=[ 1385], 99.95th=[ 1811],
     | 99.99th=[ 2835]
   bw (  KiB/s): min=81704, max=124168, per=100.00%, avg=106388.09, stdev=1196.03, samples=959
   iops        : min=20426, max=31042, avg=26596.98, stdev=299.01, samples=959
  lat (usec)   : 250=39.03%, 500=56.44%, 750=3.64%, 1000=0.47%
  lat (msec)   : 2=0.38%, 4=0.03%, 10=0.01%
  cpu          : usr=1.14%, sys=4.75%, ctx=1596524, majf=0, minf=83
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1595938,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
   READ: bw=104MiB/s (109MB/s), 104MiB/s-104MiB/s (109MB/s-109MB/s), io=6234MiB (6537MB), run=60003-60003msec
Disk stats (read/write):
  sdb: ios=1592684/0, merge=0/0, ticks=421088/0, in_queue=421088, util=99.90%

Result 2

read_job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.16
Starting 8 processes
Jobs: 8 (f=8): [r(8)][100.0%][r=147MiB/s][r=37.6k IOPS][eta 00m:00s]
read_job: (groupid=0, jobs=8): err= 0: pid=2133: Mon Jun  5 08:23:43 2023
  read: IOPS=36.8k, BW=144MiB/s (151MB/s)(8635MiB/60002msec)
    clat (usec): min=111, max=7650, avg=215.96, stdev=69.14
     lat (usec): min=111, max=7650, avg=216.06, stdev=69.14
    clat percentiles (usec):
     |  1.00th=[  147],  5.00th=[  161], 10.00th=[  167], 20.00th=[  180],
     | 30.00th=[  188], 40.00th=[  196], 50.00th=[  204], 60.00th=[  212],
     | 70.00th=[  225], 80.00th=[  241], 90.00th=[  273], 95.00th=[  306],
     | 99.00th=[  416], 99.50th=[  502], 99.90th=[  963], 99.95th=[ 1287],
     | 99.99th=[ 2212]
   bw (  KiB/s): min=135240, max=159640, per=100.00%, avg=147356.99, stdev=620.30, samples=958
   iops        : min=33810, max=39910, avg=36839.16, stdev=155.07, samples=958
  lat (usec)   : 250=84.00%, 500=15.49%, 750=0.31%, 1000=0.11%
  lat (msec)   : 2=0.08%, 4=0.01%, 10=0.01%
  cpu          : usr=1.31%, sys=5.45%, ctx=2212010, majf=0, minf=94
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2210533,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=144MiB/s (151MB/s), 144MiB/s-144MiB/s (151MB/s-151MB/s), io=8635MiB (9054MB), run=60002-60002msec

Disk stats (read/write):
  sdb: ios=2206466/4, merge=0/1, ticks=434185/9, in_queue=434193, util=99.89%

GCP local SSD

GCP allows using local SSD which suppose to have better performance than the persistent one.
Benchmark result shows better latency when numjobs=1, but for some reason it performs much worse with parallel requests.

Result with numjobs=1

read_job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.1
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=30.5MiB/s,w=0KiB/s][r=7813,w=0 IOPS][eta 00m:00s]
read_job: (groupid=0, jobs=1): err= 0: pid=24746: Mon Jun  5 08:21:47 2023
   read: IOPS=7953, BW=31.1MiB/s (32.6MB/s)(1864MiB/60001msec)
    clat (usec): min=107, max=1293, avg=125.14, stdev=10.73
     lat (usec): min=107, max=1293, avg=125.21, stdev=10.73
    clat percentiles (usec):
     |  1.00th=[  116],  5.00th=[  118], 10.00th=[  119], 20.00th=[  121],
     | 30.00th=[  122], 40.00th=[  123], 50.00th=[  123], 60.00th=[  124],
     | 70.00th=[  125], 80.00th=[  127], 90.00th=[  133], 95.00th=[  145],
     | 99.00th=[  167], 99.50th=[  174], 99.90th=[  210], 99.95th=[  258],
     | 99.99th=[  334]
   bw (  KiB/s): min=30488, max=33656, per=100.00%, avg=31822.41, stdev=697.57, samples=119
   iops        : min= 7622, max= 8414, avg=7955.61, stdev=174.37, samples=119
  lat (usec)   : 250=99.95%, 500=0.05%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%
  cpu          : usr=1.24%, sys=9.36%, ctx=477228, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=477208,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
   READ: bw=31.1MiB/s (32.6MB/s), 31.1MiB/s-31.1MiB/s (32.6MB/s-32.6MB/s), io=1864MiB (1955MB), run=60001-60001msec
Disk stats (read/write):
  nvme0n1: ios=476426/0, merge=0/0, ticks=56139/0, in_queue=0, util=99.90%

Result with numjobs=4

read_job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.1
Starting 4 processes
Jobs: 4 (f=4): [r(4)][100.0%][r=87.9MiB/s,w=0KiB/s][r=22.5k,w=0 IOPS][eta 00m:00s]
read_job: (groupid=0, jobs=4): err= 0: pid=26617: Mon Jun  5 08:52:43 2023
   read: IOPS=22.5k, BW=87.8MiB/s (92.1MB/s)(5268MiB/60001msec)
    clat (usec): min=114, max=3644, avg=176.76, stdev=17.26
     lat (usec): min=114, max=3644, avg=176.88, stdev=17.26
    clat percentiles (usec):
     |  1.00th=[  143],  5.00th=[  161], 10.00th=[  165], 20.00th=[  172],
     | 30.00th=[  174], 40.00th=[  176], 50.00th=[  176], 60.00th=[  176],
     | 70.00th=[  178], 80.00th=[  182], 90.00th=[  188], 95.00th=[  196],
     | 99.00th=[  223], 99.50th=[  241], 99.90th=[  424], 99.95th=[  449],
     | 99.99th=[  519]
   bw (  KiB/s): min=21264, max=22712, per=25.00%, avg=22476.22, stdev=192.75, samples=476
   iops        : min= 5316, max= 5678, avg=5619.04, stdev=48.18, samples=476
  lat (usec)   : 250=99.61%, 500=0.38%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%
  cpu          : usr=1.51%, sys=8.24%, ctx=1348596, majf=0, minf=38
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=1348536,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
   READ: bw=87.8MiB/s (92.1MB/s), 87.8MiB/s-87.8MiB/s (92.1MB/s-92.1MB/s), io=5268MiB (5524MB), run=60001-60001msec
Disk stats (read/write):
  nvme0n1: ios=1346082/0, merge=0/0, ticks=224692/0, in_queue=0, util=99.89%

Bare metal SSD

MacBook Pro 2021 laptop with 512GB SSD was used to test SSD performance in bare metal setup.

Result

read_job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1                                                        
...                                                                                                                                                                  
fio-3.33                                                                                                                                                             
Starting 8 processes                                                                                                                                                 
Jobs: 8 (f=8): [r(8)][100.0%][r=760MiB/s][r=194k IOPS][eta 00m:00s]                                                                                                  
read_job: (groupid=0, jobs=8): err= 0: pid=52799: Mon Jun  5 10:53:50 2023                                                                                           
  read: IOPS=235k, BW=917MiB/s (961MB/s)(53.7GiB/60001msec)                                                                                                          
    clat (usec): min=15, max=8874, avg=33.52, stdev=14.90                                                                                                            
     lat (usec): min=15, max=8874, avg=33.57, stdev=14.93                                                                                                            
    clat percentiles (usec):                                                                                                                                         
     |  1.00th=[   22],  5.00th=[   23], 10.00th=[   24], 20.00th=[   25],                                                                                           
     | 30.00th=[   30], 40.00th=[   33], 50.00th=[   34], 60.00th=[   35],                                                                                           
     | 70.00th=[   37], 80.00th=[   39], 90.00th=[   43], 95.00th=[   48],                                                                                           
     | 99.00th=[   64], 99.50th=[   71], 99.90th=[  101], 99.95th=[  133],                                                                                           
     | 99.99th=[  371]                                                                                                                                               
   bw (  KiB/s): min=689636, max=1026288, per=100.00%, avg=940659.13, stdev=11858.08, samples=952                                                                    
   iops        : min=172406, max=256572, avg=235163.07, stdev=2964.56, samples=952                                                                                   
  lat (usec)   : 20=0.05%, 50=95.69%, 100=4.16%, 250=0.09%, 500=0.01%                                                                                                
  lat (usec)   : 750=0.01%, 1000=0.01%                                                                                                                               
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%
  cpu          : usr=2.13%, sys=24.31%, ctx=18217674, majf=0, minf=78
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=14081057,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=917MiB/s (961MB/s), 917MiB/s-917MiB/s (961MB/s-961MB/s), io=53.7GiB (57.7GB), run=60001-60001msec

Results Summary

Metric	GCP persistent	GCP local	Bare metal
avg latency	200-300us	125us	34us
parallelism	8	2-3	8
IOPS	25-40k	22k	194k

Longarithm · 2023-07-25T17:11:26Z

Longarithm
Jul 25, 2023
Maintainer

From discussion with @pugachAG - we also tried more threads (numjobs) but parallelism for persistent SSD stayed at 8.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Raw SSD performance benchmark #9147

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment

{{title}}

Select a reply

Raw SSD performance benchmark #9147

pugachAG Jun 6, 2023 Collaborator

Setup

GCP persistant SSD

GCP local SSD

Bare metal SSD

Results Summary

Replies: 1 comment

Longarithm Jul 25, 2023 Maintainer

pugachAG
Jun 6, 2023
Collaborator

Longarithm
Jul 25, 2023
Maintainer