How to Access NVMe disk Using SPDK When Creating a DiskPool

[Hansyangxingxiang]

How to Access NVMe disk Using SPDK When Creating a DiskPool ？
In the documentation under the "type" section, I only see the following:

Type Format Example
Disk (non PCI) with disk-by-guid reference (Best Practice):
Device File: aio:///dev/disk/by-id/ OR uring:///dev/disk/by-id/
Asynchronous Disk (AIO):
Device File: /dev/sdx
Device File (AIO): aio:///dev/sdx
io_uring:
Device File: uring:///dev/sdx
However, there is no mention of how to use SPDK.

Could you provide guidance on how to configure a DiskPool to use SPDK for accessing NVMe drives?

[tiagolobocastro]

You can connect in different ways:

1. Via a Kernel BDev
   a. URING: uring:///dev/nvmexn1
   b. AIO: uring:///dev/nvmexn1 or simply /dev/nvmexn1
2. Direct PCIe access
   a. pcie:///0000:01:00.0

If you want Direct PCIe access (2.a) bypassing the kernel then a bit more work is involved (though we plan to automate this in the future - it's on the ROADMAP) as you have to detach the nvme device from the kernel.

You'll have to use https://github.com/spdk/spdk/blob/master/scripts/setup.sh, probably need to clone the spdk repo.

❯ sudo ./scripts/setup.sh status
Hugepages
node     hugesize     free /  total
node0   1048576kB        0 /      0
node0      2048kB     4096 /   4096

Type                      BDF             Vendor Device NUMA    Driver           Device     Block devices
NVMe                      0000:01:00.0    144d   a80c   unknown nvme             nvme0      nvme0n1
NVMe                      0000:22:00.0    8086   f1a8   unknown nvme             nvme1      nvme1n1
NVMe                      0000:23:00.0    144d   a808   unknown nvme             nvme2      nvme2n1

In my case I want to use pcie for nvme0n1, which has BDF of 0000:01:00.0

❯ sudo PCI_ALLOWED=0000:01:00.0 ./scripts/setup.sh
....
❯ sudo ./scripts/setup.sh status
Hugepages
node     hugesize     free /  total
node0   1048576kB        0 /      0
node0      2048kB     4096 /   4096

Type                      BDF             Vendor Device NUMA    Driver           Device     Block devices
NVMe                      0000:01:00.0    144d   a80c   unknown vfio-pci         -          -
NVMe                      0000:22:00.0    8086   f1a8   unknown nvme             nvme1      nvme1n1
NVMe                      0000:23:00.0    144d   a808   unknown nvme             nvme2      nvme2n1

Now that 0000:01:00.0 is bound to vfio-pcie, now you can create your pools with pcie:///0000:01:00.0.
As you can see atm this is not very straightforward, so I suggest you try aio or uring first.

Hope this helps!

[orville-wright]

@avishnu : We should have @balaharish7 integrate this answer into the Tech Docs. - This is very good info.

[balaharish7]

Team, this is documented in PR #455