Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

vastly reduced speeds using six NVMe raidz1 as plain, compression, encryption, or both on 0.8.3-1ubuntu3 #10066

Closed
sneak opened this issue Feb 27, 2020 · 2 comments
Closed

vastly reduced speeds using six NVMe raidz1 as plain, compression, encryption, or both on 0.8.3-1ubuntu3 #10066

sneak opened this issue Feb 27, 2020 · 2 comments
Labels
Status: Stale No recent activity for issue

Comments

@sneak
Copy link

sneak commented Feb 27, 2020 •
edited
Loading

Note: This may be related to #9786 or #8836, I am not sure.

System information

Type Version/Name
Distribution Name Ubuntu
Distribution Version 20.04 focal (development)
Linux Kernel 5.4.0-14-generic #17-Ubuntu
Architecture amd64
ZFS Version 0.8.3-1ubuntu3
SPL Version 0.8.3-1ubuntu3
System CPU AMD EPYC 7401P 24-Core Processor
System Cores 24
System Threads 48
System Memory 256G
AES-NI Yes

Describe the problem you're observing

Six disk raidz1 of NVMe Enterprise 960GB SSDs. Each SSD can get about 700MB/s with a single thread or 3000MB/s with 16 threads (which is itself weird to me, but this is my first time using NVMe on linux).

Single-disk example:

testing /dev/disk/by-id/nvme-SAMSUNG_MZQLB960HAJR-00007_S437NE0M802527

filename=/dev/disk/by-id/nvme-SAMSUNG_MZQLB960HAJR-00007_S437NE0M802527
test-64kchunk-1thread: (g=0): rw=rw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
  read: IOPS=11.1k, BW=693MiB/s (727MB/s)(40.6GiB/60001msec)
  write: IOPS=11.1k, BW=693MiB/s (726MB/s)(40.6GiB/60001msec); 0 zone resets
   READ: bw=693MiB/s (727MB/s), 693MiB/s-693MiB/s (727MB/s-727MB/s), io=40.6GiB (43.6GB), run=60001-60001msec
  WRITE: bw=693MiB/s (726MB/s), 693MiB/s-693MiB/s (726MB/s-726MB/s), io=40.6GiB (43.6GB), run=60001-60001msec

filename=/dev/disk/by-id/nvme-SAMSUNG_MZQLB960HAJR-00007_S437NE0M802527
test-64kchunk-16thread: (g=0): rw=rw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
  read: IOPS=49.8k, BW=3111MiB/s (3263MB/s)(185GiB/60935msec)
  write: IOPS=49.8k, BW=3116MiB/s (3267MB/s)(185GiB/60935msec); 0 zone resets
   READ: bw=3111MiB/s (3263MB/s), 3111MiB/s-3111MiB/s (3263MB/s-3263MB/s), io=185GiB (199GB), run=60935-60935msec
  WRITE: bw=3116MiB/s (3267MB/s), 3116MiB/s-3116MiB/s (3267MB/s-3267MB/s), io=185GiB (199GB), run=60935-60935msec

In raidz1 (using the whole disk) I have created child datasets like so:

+ zfs create tank/none
+ zfs create -o compression=lz4 tank/comp
+ zfs create -o encryption=aes-256-gcm -o keyformat=raw -o keylocation=file:///tmp/testkey.raw tank/enc
+ zfs create -o compression=lz4 -o encryption=aes-256-gcm -o keyformat=raw -o keylocation=file:///tmp/testkey.raw tank/both

I then test tank/none, tank/comp (compression), tank/enc (encryption), tank/both (compression and encryption). I repeated this test with both 1 process and 16 processes, and then formatted the NVMs and repeated with ashift=12,13,14,15,16.

Test settings: iodepth=64, ioengine=libaio, rw=rw, numjobs={1,16},bs=64k,runtime=60.

The results are as follows. Especially surprising are 1099MB/s speeds using just compression across a six disk NVMe on a 24 core machine when a single disk can do 3000MB/s, as well as 35MB/s (!) speeds using encryption. I am also confused as to why a single process can't seem to max out even the single disk bandwidth when there is CPU to spare.

Also confusing to me is why a six-disk raidz1 of 3000MB/s disks can only seem to work at 2700MB/s maximum.

ashift=12

test-none-64kchunk-1thread: (g=0): rw=rw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
test-none-64kchunk-1thread: (groupid=0, jobs=1): err= 0: pid=45480: Wed Feb 26 09:35:40 2020
   READ: bw=1061MiB/s (1112MB/s), 1061MiB/s-1061MiB/s (1112MB/s-1112MB/s), io=124GiB (133GB), run=120001-120001msec
  WRITE: bw=1059MiB/s (1111MB/s), 1059MiB/s-1059MiB/s (1111MB/s-1111MB/s), io=124GiB (133GB), run=120001-120001msec

test-comp-64kchunk-1thread: (g=0): rw=rw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
test-comp-64kchunk-1thread: (groupid=0, jobs=1): err= 0: pid=329592: Wed Feb 26 09:37:52 2020
   READ: bw=1057MiB/s (1108MB/s), 1057MiB/s-1057MiB/s (1108MB/s-1108MB/s), io=124GiB (133GB), run=120001-120001msec
  WRITE: bw=1055MiB/s (1107MB/s), 1055MiB/s-1055MiB/s (1107MB/s-1107MB/s), io=124GiB (133GB), run=120001-120001msec

test-enc-64kchunk-1thread: (g=0): rw=rw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
test-enc-64kchunk-1thread: (groupid=0, jobs=1): err= 0: pid=549276: Wed Feb 26 09:40:05 2020
   READ: bw=34.9MiB/s (36.6MB/s), 34.9MiB/s-34.9MiB/s (36.6MB/s-36.6MB/s), io=4191MiB (4395MB), run=120002-120002msec
  WRITE: bw=34.8MiB/s (36.5MB/s), 34.8MiB/s-34.8MiB/s (36.5MB/s-36.5MB/s), io=4172MiB (4374MB), run=120002-120002msec

test-both-64kchunk-1thread: (g=0): rw=rw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
test-both-64kchunk-1thread: (groupid=0, jobs=1): err= 0: pid=660255: Wed Feb 26 09:42:18 2020
   READ: bw=34.9MiB/s (36.6MB/s), 34.9MiB/s-34.9MiB/s (36.6MB/s-36.6MB/s), io=4187MiB (4390MB), run=120004-120004msec
  WRITE: bw=34.7MiB/s (36.4MB/s), 34.7MiB/s-34.7MiB/s (36.4MB/s-36.4MB/s), io=4167MiB (4370MB), run=120004-120004msec

test-none-64kchunk-16thread: (g=0): rw=rw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
test-none-64kchunk-16thread: (groupid=0, jobs=16): err= 0: pid=951511: Wed Feb 26 09:47:03 2020
   READ: bw=2612MiB/s (2739MB/s), 2612MiB/s-2612MiB/s (2739MB/s-2739MB/s), io=306GiB (329GB), run=120006-120006msec
  WRITE: bw=2614MiB/s (2741MB/s), 2614MiB/s-2614MiB/s (2741MB/s-2741MB/s), io=306GiB (329GB), run=120006-120006msec

test-comp-64kchunk-16thread: (g=0): rw=rw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
test-comp-64kchunk-16thread: (groupid=0, jobs=16): err= 0: pid=1584172: Wed Feb 26 09:51:53 2020
   READ: bw=2592MiB/s (2718MB/s), 2592MiB/s-2592MiB/s (2718MB/s-2718MB/s), io=304GiB (326GB), run=120005-120005msec
  WRITE: bw=2594MiB/s (2720MB/s), 2594MiB/s-2594MiB/s (2720MB/s-2720MB/s), io=304GiB (326GB), run=120005-120005msec

test-enc-64kchunk-16thread: (g=0): rw=rw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
test-enc-64kchunk-16thread: (groupid=0, jobs=16): err= 0: pid=3698924: Wed Feb 26 10:14:54 2020
   READ: bw=348MiB/s (364MB/s), 348MiB/s-348MiB/s (364MB/s-364MB/s), io=40.7GiB (43.7GB), run=120008-120008msec
  WRITE: bw=347MiB/s (364MB/s), 347MiB/s-347MiB/s (364MB/s-364MB/s), io=40.7GiB (43.7GB), run=120008-120008msec

test-both-64kchunk-16thread: (g=0): rw=rw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
test-both-64kchunk-16thread: (groupid=0, jobs=16): err= 0: pid=1492932: Wed Feb 26 10:37:54 2020
   READ: bw=331MiB/s (347MB/s), 331MiB/s-331MiB/s (347MB/s-347MB/s), io=38.8GiB (41.6GB), run=120008-120008msec
  WRITE: bw=331MiB/s (347MB/s), 331MiB/s-331MiB/s (347MB/s-347MB/s), io=38.8GiB (41.6GB), run=120008-120008msec

ashift=13

test-none-64kchunk-1thread: (g=0): rw=rw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
test-none-64kchunk-1thread: (groupid=0, jobs=1): err= 0: pid=1722082: Wed Feb 26 10:41:19 2020
   READ: bw=1105MiB/s (1159MB/s), 1105MiB/s-1105MiB/s (1159MB/s-1159MB/s), io=130GiB (139GB), run=120001-120001msec
  WRITE: bw=1104MiB/s (1157MB/s), 1104MiB/s-1104MiB/s (1157MB/s-1157MB/s), io=129GiB (139GB), run=120001-120001msec

test-comp-64kchunk-1thread: (g=0): rw=rw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
test-comp-64kchunk-1thread: (groupid=0, jobs=1): err= 0: pid=2023255: Wed Feb 26 10:43:30 2020
   READ: bw=1075MiB/s (1127MB/s), 1075MiB/s-1075MiB/s (1127MB/s-1127MB/s), io=126GiB (135GB), run=120001-120001msec
  WRITE: bw=1074MiB/s (1126MB/s), 1074MiB/s-1074MiB/s (1126MB/s-1126MB/s), io=126GiB (135GB), run=120001-120001msec

test-enc-64kchunk-1thread: (g=0): rw=rw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
test-enc-64kchunk-1thread: (groupid=0, jobs=1): err= 0: pid=2232693: Wed Feb 26 10:45:44 2020
   READ: bw=34.9MiB/s (36.6MB/s), 34.9MiB/s-34.9MiB/s (36.6MB/s-36.6MB/s), io=4190MiB (4393MB), run=120003-120003msec
  WRITE: bw=34.8MiB/s (36.4MB/s), 34.8MiB/s-34.8MiB/s (36.4MB/s-36.4MB/s), io=4171MiB (4374MB), run=120003-120003msec

test-both-64kchunk-1thread: (g=0): rw=rw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
test-both-64kchunk-1thread: (groupid=0, jobs=1): err= 0: pid=2344499: Wed Feb 26 10:47:57 2020
   READ: bw=34.9MiB/s (36.6MB/s), 34.9MiB/s-34.9MiB/s (36.6MB/s-36.6MB/s), io=4187MiB (4391MB), run=120003-120003msec
  WRITE: bw=34.7MiB/s (36.4MB/s), 34.7MiB/s-34.7MiB/s (36.4MB/s-36.4MB/s), io=4168MiB (4371MB), run=120003-120003msec

test-none-64kchunk-16thread: (g=0): rw=rw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
test-none-64kchunk-16thread: (groupid=0, jobs=16): err= 0: pid=2594427: Wed Feb 26 10:52:39 2020
   READ: bw=2856MiB/s (2994MB/s), 2856MiB/s-2856MiB/s (2994MB/s-2994MB/s), io=335GiB (359GB), run=120002-120002msec
  WRITE: bw=2858MiB/s (2997MB/s), 2858MiB/s-2858MiB/s (2997MB/s-2997MB/s), io=335GiB (360GB), run=120002-120002msec

test-comp-64kchunk-16thread: (g=0): rw=rw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
test-comp-64kchunk-16thread: (groupid=0, jobs=16): err= 0: pid=3252014: Wed Feb 26 10:57:26 2020
   READ: bw=2446MiB/s (2565MB/s), 2446MiB/s-2446MiB/s (2565MB/s-2565MB/s), io=287GiB (308GB), run=120005-120005msec
  WRITE: bw=2448MiB/s (2567MB/s), 2448MiB/s-2448MiB/s (2567MB/s-2567MB/s), io=287GiB (308GB), run=120005-120005msec

test-enc-64kchunk-16thread: (g=0): rw=rw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
test-enc-64kchunk-16thread: (groupid=0, jobs=16): err= 0: pid=1064712: Wed Feb 26 11:20:24 2020
   READ: bw=339MiB/s (356MB/s), 339MiB/s-339MiB/s (356MB/s-356MB/s), io=39.8GiB (42.7GB), run=120017-120017msec
  WRITE: bw=339MiB/s (356MB/s), 339MiB/s-339MiB/s (356MB/s-356MB/s), io=39.8GiB (42.7GB), run=120017-120017msec

test-both-64kchunk-16thread: (g=0): rw=rw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
test-both-64kchunk-16thread: (groupid=0, jobs=16): err= 0: pid=3025466: Wed Feb 26 11:43:26 2020
   READ: bw=345MiB/s (362MB/s), 345MiB/s-345MiB/s (362MB/s-362MB/s), io=40.4GiB (43.4GB), run=120013-120013msec
  WRITE: bw=345MiB/s (362MB/s), 345MiB/s-345MiB/s (362MB/s-362MB/s), io=40.4GiB (43.4GB), run=120013-120013msec

ashift=14

test-none-64kchunk-1thread: (g=0): rw=rw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
test-none-64kchunk-1thread: (groupid=0, jobs=1): err= 0: pid=3266552: Wed Feb 26 11:46:52 2020
   READ: bw=1086MiB/s (1138MB/s), 1086MiB/s-1086MiB/s (1138MB/s-1138MB/s), io=127GiB (137GB), run=120001-120001msec
  WRITE: bw=1084MiB/s (1137MB/s), 1084MiB/s-1084MiB/s (1137MB/s-1137MB/s), io=127GiB (136GB), run=120001-120001msec

test-comp-64kchunk-1thread: (g=0): rw=rw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
test-comp-64kchunk-1thread: (groupid=0, jobs=1): err= 0: pid=3532800: Wed Feb 26 11:49:03 2020
   READ: bw=1059MiB/s (1110MB/s), 1059MiB/s-1059MiB/s (1110MB/s-1110MB/s), io=124GiB (133GB), run=120001-120001msec
  WRITE: bw=1058MiB/s (1109MB/s), 1058MiB/s-1058MiB/s (1109MB/s-1109MB/s), io=124GiB (133GB), run=120001-120001msec

test-enc-64kchunk-1thread: (g=0): rw=rw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
test-enc-64kchunk-1thread: (groupid=0, jobs=1): err= 0: pid=3746836: Wed Feb 26 11:51:17 2020
   READ: bw=34.0MiB/s (36.7MB/s), 34.0MiB/s-34.0MiB/s (36.7MB/s-36.7MB/s), io=4199MiB (4403MB), run=120002-120002msec
  WRITE: bw=34.8MiB/s (36.5MB/s), 34.8MiB/s-34.8MiB/s (36.5MB/s-36.5MB/s), io=4180MiB (4383MB), run=120002-120002msec

test-both-64kchunk-1thread: (g=0): rw=rw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
test-both-64kchunk-1thread: (groupid=0, jobs=1): err= 0: pid=3857017: Wed Feb 26 11:53:30 2020
   READ: bw=34.9MiB/s (36.6MB/s), 34.9MiB/s-34.9MiB/s (36.6MB/s-36.6MB/s), io=4190MiB (4393MB), run=120002-120002msec
  WRITE: bw=34.8MiB/s (36.4MB/s), 34.8MiB/s-34.8MiB/s (36.4MB/s-36.4MB/s), io=4171MiB (4374MB), run=120002-120002msec

test-none-64kchunk-16thread: (g=0): rw=rw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
test-none-64kchunk-16thread: (groupid=0, jobs=16): err= 0: pid=4100481: Wed Feb 26 11:58:12 2020
   READ: bw=2756MiB/s (2890MB/s), 2756MiB/s-2756MiB/s (2890MB/s-2890MB/s), io=323GiB (347GB), run=120014-120014msec
  WRITE: bw=2758MiB/s (2892MB/s), 2758MiB/s-2758MiB/s (2892MB/s-2892MB/s), io=323GiB (347GB), run=120014-120014msec

test-comp-64kchunk-16thread: (g=0): rw=rw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
test-comp-64kchunk-16thread: (groupid=0, jobs=16): err= 0: pid=518136: Wed Feb 26 12:02:56 2020
   READ: bw=2733MiB/s (2866MB/s), 2733MiB/s-2733MiB/s (2866MB/s-2866MB/s), io=320GiB (344GB), run=120001-120001msec
  WRITE: bw=2735MiB/s (2868MB/s), 2735MiB/s-2735MiB/s (2868MB/s-2868MB/s), io=321GiB (344GB), run=120001-120001msec

test-enc-64kchunk-16thread: (g=0): rw=rw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
test-enc-64kchunk-16thread: (groupid=0, jobs=16): err= 0: pid=2570477: Wed Feb 26 12:25:43 2020
   READ: bw=351MiB/s (368MB/s), 351MiB/s-351MiB/s (368MB/s-368MB/s), io=41.1GiB (44.2GB), run=120010-120010msec
  WRITE: bw=351MiB/s (368MB/s), 351MiB/s-351MiB/s (368MB/s-368MB/s), io=41.1GiB (44.2GB), run=120010-120010msec

test-both-64kchunk-16thread: (g=0): rw=rw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
test-both-64kchunk-16thread: (groupid=0, jobs=16): err= 0: pid=312314: Wed Feb 26 12:48:47 2020
   READ: bw=332MiB/s (348MB/s), 332MiB/s-332MiB/s (348MB/s-348MB/s), io=38.9GiB (41.8GB), run=120009-120009msec
  WRITE: bw=332MiB/s (348MB/s), 332MiB/s-332MiB/s (348MB/s-348MB/s), io=38.9GiB (41.8GB), run=120009-120009msec

ashift=15

test-none-64kchunk-1thread: (g=0): rw=rw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
test-none-64kchunk-1thread: (groupid=0, jobs=1): err= 0: pid=541817: Wed Feb 26 12:52:13 2020
   READ: bw=1103MiB/s (1156MB/s), 1103MiB/s-1103MiB/s (1156MB/s-1156MB/s), io=129GiB (139GB), run=120001-120001msec
  WRITE: bw=1101MiB/s (1155MB/s), 1101MiB/s-1101MiB/s (1155MB/s-1155MB/s), io=129GiB (139GB), run=120001-120001msec

test-comp-64kchunk-1thread: (g=0): rw=rw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
test-comp-64kchunk-1thread: (groupid=0, jobs=1): err= 0: pid=797574: Wed Feb 26 12:54:25 2020
   READ: bw=1078MiB/s (1131MB/s), 1078MiB/s-1078MiB/s (1131MB/s-1131MB/s), io=126GiB (136GB), run=120001-120001msec
  WRITE: bw=1077MiB/s (1129MB/s), 1077MiB/s-1077MiB/s (1129MB/s-1129MB/s), io=126GiB (136GB), run=120001-120001msec

test-enc-64kchunk-1thread: (g=0): rw=rw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
test-enc-64kchunk-1thread: (groupid=0, jobs=1): err= 0: pid=1005711: Wed Feb 26 12:56:38 2020
   READ: bw=35.0MiB/s (36.7MB/s), 35.0MiB/s-35.0MiB/s (36.7MB/s-36.7MB/s), io=4200MiB (4404MB), run=120004-120004msec
  WRITE: bw=34.8MiB/s (36.5MB/s), 34.8MiB/s-34.8MiB/s (36.5MB/s-36.5MB/s), io=4182MiB (4385MB), run=120004-120004msec

test-both-64kchunk-1thread: (g=0): rw=rw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
test-both-64kchunk-1thread: (groupid=0, jobs=1): err= 0: pid=1104483: Wed Feb 26 12:58:51 2020
   READ: bw=34.0MiB/s (36.7MB/s), 34.0MiB/s-34.0MiB/s (36.7MB/s-36.7MB/s), io=4199MiB (4403MB), run=120002-120002msec
  WRITE: bw=34.8MiB/s (36.5MB/s), 34.8MiB/s-34.8MiB/s (36.5MB/s-36.5MB/s), io=4181MiB (4384MB), run=120002-120002msec

test-none-64kchunk-16thread: (g=0): rw=rw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
test-none-64kchunk-16thread: (groupid=0, jobs=16): err= 0: pid=1346872: Wed Feb 26 13:03:30 2020
   READ: bw=2792MiB/s (2927MB/s), 2792MiB/s-2792MiB/s (2927MB/s-2927MB/s), io=327GiB (351GB), run=120002-120002msec
  WRITE: bw=2794MiB/s (2929MB/s), 2794MiB/s-2794MiB/s (2929MB/s-2929MB/s), io=327GiB (352GB), run=120002-120002msec

test-comp-64kchunk-16thread: (g=0): rw=rw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
test-comp-64kchunk-16thread: (groupid=0, jobs=16): err= 0: pid=1944480: Wed Feb 26 13:08:13 2020
   READ: bw=2745MiB/s (2879MB/s), 2745MiB/s-2745MiB/s (2879MB/s-2879MB/s), io=322GiB (345GB), run=120006-120006msec
  WRITE: bw=2747MiB/s (2881MB/s), 2747MiB/s-2747MiB/s (2881MB/s-2881MB/s), io=322GiB (346GB), run=120006-120006msec

test-enc-64kchunk-16thread: (g=0): rw=rw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
test-enc-64kchunk-16thread: (groupid=0, jobs=16): err= 0: pid=3808807: Wed Feb 26 13:31:06 2020
   READ: bw=352MiB/s (369MB/s), 352MiB/s-352MiB/s (369MB/s-369MB/s), io=41.2GiB (44.3GB), run=120007-120007msec
  WRITE: bw=352MiB/s (369MB/s), 352MiB/s-352MiB/s (369MB/s-369MB/s), io=41.2GiB (44.3GB), run=120007-120007msec

test-both-64kchunk-16thread: (g=0): rw=rw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
test-both-64kchunk-16thread: (groupid=0, jobs=16): err= 0: pid=1382283: Wed Feb 26 13:54:05 2020
   READ: bw=339MiB/s (355MB/s), 339MiB/s-339MiB/s (355MB/s-355MB/s), io=39.7GiB (42.6GB), run=120021-120021msec
  WRITE: bw=339MiB/s (355MB/s), 339MiB/s-339MiB/s (355MB/s-355MB/s), io=39.7GiB (42.6GB), run=120021-120021msec

ashift=16

test-none-64kchunk-1thread: (g=0): rw=rw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
test-none-64kchunk-1thread: (groupid=0, jobs=1): err= 0: pid=1612261: Wed Feb 26 13:57:29 2020
   READ: bw=1145MiB/s (1201MB/s), 1145MiB/s-1145MiB/s (1201MB/s-1201MB/s), io=134GiB (144GB), run=120001-120001msec
  WRITE: bw=1144MiB/s (1200MB/s), 1144MiB/s-1144MiB/s (1200MB/s-1200MB/s), io=134GiB (144GB), run=120001-120001msec

test-comp-64kchunk-1thread: (g=0): rw=rw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
test-comp-64kchunk-1thread: (groupid=0, jobs=1): err= 0: pid=2006008: Wed Feb 26 13:59:41 2020
   READ: bw=1099MiB/s (1152MB/s), 1099MiB/s-1099MiB/s (1152MB/s-1152MB/s), io=129GiB (138GB), run=120001-120001msec
  WRITE: bw=1097MiB/s (1151MB/s), 1097MiB/s-1097MiB/s (1151MB/s-1151MB/s), io=129GiB (138GB), run=120001-120001msec

test-enc-64kchunk-1thread: (g=0): rw=rw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
test-enc-64kchunk-1thread: (groupid=0, jobs=1): err= 0: pid=2261607: Wed Feb 26 14:01:54 2020
   READ: bw=35.1MiB/s (36.8MB/s), 35.1MiB/s-35.1MiB/s (36.8MB/s-36.8MB/s), io=4210MiB (4414MB), run=120002-120002msec
  WRITE: bw=34.9MiB/s (36.6MB/s), 34.9MiB/s-34.9MiB/s (36.6MB/s-36.6MB/s), io=4190MiB (4394MB), run=120002-120002msec

test-both-64kchunk-1thread: (g=0): rw=rw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
test-both-64kchunk-1thread: (groupid=0, jobs=1): err= 0: pid=2326082: Wed Feb 26 14:04:07 2020
   READ: bw=35.2MiB/s (36.9MB/s), 35.2MiB/s-35.2MiB/s (36.9MB/s-36.9MB/s), io=4225MiB (4430MB), run=120002-120002msec
  WRITE: bw=35.0MiB/s (36.7MB/s), 35.0MiB/s-35.0MiB/s (36.7MB/s-36.7MB/s), io=4204MiB (4408MB), run=120002-120002msec

test-none-64kchunk-16thread: (g=0): rw=rw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
test-none-64kchunk-16thread: (groupid=0, jobs=16): err= 0: pid=2595459: Wed Feb 26 14:08:48 2020
   READ: bw=2780MiB/s (2915MB/s), 2780MiB/s-2780MiB/s (2915MB/s-2915MB/s), io=326GiB (350GB), run=120002-120002msec
  WRITE: bw=2782MiB/s (2917MB/s), 2782MiB/s-2782MiB/s (2917MB/s-2917MB/s), io=326GiB (350GB), run=120002-120002msec

test-comp-64kchunk-16thread: (g=0): rw=rw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
test-comp-64kchunk-16thread: (groupid=0, jobs=16): err= 0: pid=3476111: Wed Feb 26 14:13:38 2020
   READ: bw=2757MiB/s (2891MB/s), 2757MiB/s-2757MiB/s (2891MB/s-2891MB/s), io=323GiB (347GB), run=120001-120001msec
  WRITE: bw=2760MiB/s (2894MB/s), 2760MiB/s-2760MiB/s (2894MB/s-2894MB/s), io=323GiB (347GB), run=120001-120001msec

test-enc-64kchunk-16thread: (g=0): rw=rw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
test-enc-64kchunk-16thread: (groupid=0, jobs=16): err= 0: pid=643126: Wed Feb 26 14:36:19 2020
   READ: bw=363MiB/s (380MB/s), 363MiB/s-363MiB/s (380MB/s-380MB/s), io=42.5GiB (45.7GB), run=120006-120006msec
  WRITE: bw=363MiB/s (380MB/s), 363MiB/s-363MiB/s (380MB/s-380MB/s), io=42.5GiB (45.7GB), run=120006-120006msec

test-both-64kchunk-16thread: (g=0): rw=rw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
test-both-64kchunk-16thread: (groupid=0, jobs=16): err= 0: pid=1765030: Wed Feb 26 14:59:08 2020
   READ: bw=358MiB/s (376MB/s), 358MiB/s-358MiB/s (376MB/s-376MB/s), io=42.0GiB (45.1GB), run=120011-120011msec
  WRITE: bw=358MiB/s (376MB/s), 358MiB/s-358MiB/s (376MB/s-376MB/s), io=42.0GiB (45.1GB), run=120011-120011msec

The complete log output (the above lines are just a summary) can be found at:

https://gist.github.com/838611b86ac54a138f68f59ba44f10fb

Describe how to reproduce the problem

Precise steps to reproduce are located in this testing script:

https://gist.github.com/sneak/3e14a2a3fa04fdc73ca51c72a105a884

And the full test output:

https://gist.github.com/838611b86ac54a138f68f59ba44f10fb

Include any warning/errors/backtraces from the system logs

Nothing anomalous except the following:

[ 5364.173793] perf: interrupt took too long (2501 > 2500), lowering kernel.perf_event_max_sample_rate to 79750
[ 8798.394144] perf: interrupt took too long (3128 > 3126), lowering kernel.perf_event_max_sample_rate to 63750
[14382.183488] perf: interrupt took too long (3945 > 3910), lowering kernel.perf_event_max_sample_rate to 50500

Request

Further troubleshooting steps are appreciated. Should I be building/testing against master? I need to return this box to the colo if I can't get it in shape relatively soon.

I am happy to run any additional tests or experiments that anyone can think of.
@sneak sneak changed the title vastly reduced speeds using compression, encryption, or both on 0.8.3-1ubuntu3 vastly reduced speeds using six NVMe zpool as plain, compression, encryption, or both on 0.8.3-1ubuntu3 Feb 27, 2020
@sneak sneak changed the title vastly reduced speeds using six NVMe zpool as plain, compression, encryption, or both on 0.8.3-1ubuntu3 vastly reduced speeds using six NVMe raidz1 as plain, compression, encryption, or both on 0.8.3-1ubuntu3 Feb 27, 2020
@sneak
Copy link
Author

sneak commented Feb 28, 2020

I returned the machine and can no longer run tests against that configuration.

@nustiueudinastea nustiueudinastea mentioned this issue Mar 25, 2020
Open
@stale
Copy link

stale bot commented Feb 27, 2021

This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.

@stale stale bot added the Status: Stale No recent activity for issue label Feb 27, 2021
@stale stale bot closed this as completed May 28, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Status: Stale No recent activity for issue
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@sneak