RDMA SRQ implementation for NVMe-oF target

[EugeneKochetov]

nvmf/rdma: Add shared receive queue support

This is a new feature for NVMEoF RDMA target, that is intended to save resource allocation
(by sharing them) and utilize the locality (completions and memory) to get the best
performance with Shared Receive Queues (SRQs). We'll create a SRQ per core (poll group),
per device and associate each created QP/CQ with an appropriate SRQ.

Our testing environment has 2 hosts.
Host 1:
CPU: Intel(R) Xeon(R) CPU E5-2609 0 @ 2.40GHz dual socket (8 cores total)
Network: ConnectX-5, ConnectX-5 VPI , 100GbE, single-port QSFP28, PCIe3.0 x16
Disk: Intel Optane SSD 900P Series
OS: Fedora 27 x86_64
Host 2:
CPU: Intel(R) Xeon(R) CPU E5-2630 v2 @ 2.60GHz dual-socket (24 cores total)
Network: ConnectX-4 VPI , 100GbE, dual-port QSFP28
Disk: Intel Optane SSD 900P Series
OS : CentOS 7.5.1804 x86_64
Hosts are connected via Spectrum switch.
Host 1 is running SPDK NVMeoF target. Host 2 is used as initiator running fio with SPDK plugin.

Configuration:

• SPDK NVMeoF target: cpu mask 0x0F (4 cores), max queue depth 128, max SRQ depth 1024, max QPs per controller 1024
• Single NVMf subsystem with single namespace backed by physical SSD disk
• fio with SPDK plugin: randread pattern, 1-256 jobs, block size 4k, IO depth 16, cpu_mask 0xFFF0, IO rate 10k, rate process “poisson”

Here is a full fio command line:
fio --name=Job --stats=1 --group_reporting=1 --idle-prof=percpu --loops=1 --numjobs=1 --thread=1 --time_based=1 --runtime=30s --ramp_time=5s --bs=4k --size=4G --iodepth=16 --readwrite=randread --rwmixread=75 --randrepeat=1 --ioengine=spdk --direct=1 --gtod_reduce=0 --cpumask=0xFFF0 --rate_iops=10k --rate_process=poisson --filename=trtype=RDMA adrfam=IPv4 traddr=1.1.79.1 trsvcid=4420 ns=1

SPDK allocates the following entities for every work request in receive queue (shared or not): reqs (1024 bytes), recvs (96 bytes), cmds (64 bytes), cpls (16 bytes), in_capsule_buffer. All except the last one are fixed size. In capsule data size is configured to 4096.
Memory consumption calculation (target):

• Multiple SRQ: core_num * ib_devs_num * SRQ_depth * (1200 + in_capsule_data_size)
• Multiple RQ: queue_num * RQ_depth * (1200 + in_capsule_data_size)
  We ignore admin queues in calculations for simplicity.

Cases:

1. Multiple SRQ with 1024 entries:
   • Mem = 4 * 1 * 1024 * (1200 + 4096) = 20.7 MiB (Constant number – does not depend on initiators number)
2. RQ with 128 entries for 64 initiators:
   • Mem = 64 * 128 * (1200 + 4096) = 41.4 MiB

Results:
FIO_JOBS kIOPS Bandwidth, MiB/s AvgLatency, us MaxResidentSize, kiB
RQ SRQ RQ SRQ RQ SRQ RQ SRQ
1 8.623 8.623 33.7 33.7 13.89 14.03 144376 155624
2 17.3 17.3 67.4 67.4 14.03 14.1 145776 155700
4 34.5 34.5 135 135 14.15 14.23 146540 156184
8 69.1 69.1 270 270 14.64 14.49 148116 156960
16 138 138 540 540 14.84 15.38 151216 158668
32 276 276 1079 1079 16.5 16.61 157560 161936
64 513 502 2005 1960 1673.31 1612.38 170408 168440
128 535 526 2092 2054 3329.79 3344.03 195796 181524
256 571 571 2232 2233 6854.57 6873.37 246484 207856

We can see the benefit in memory consumption.

The drawback of using SRQ is a risk of RNR errors when multiple clients initiate large number of IOs simultaneously.
In "RQ per QP" this is handled by Submission Queue flow control and RNR is not possible.
This patch does not contain any changes to solve RNR issue but we see at least two options here:

• try to increase the RNR retry count to more than 0 which is now hardcoded and make it a configurable parameter.
• implement some mechanism for dynamic SRQ extension. It may scale with number of IO queues or when it reaches some threshold.