NVMf offload

[EugeneKochetov]

The first version of NVMf offload implementation in SPDK. Not buildable yet and breaks some abstractions.

[sashakot]

I think, ibv_device_attr_ex shouldn't be depend on SPDK_CONFIG_NVMF_OFFLOAD. We can reuse it for other features.
It's better to add it under
SPDK_EXTEND_OFED . Something like that.

[sashakot]

Hardware NVMe-oF target offload

Introduction

NVMe-over-Fabrics target offload allows an HCA to offload the complete NVMe-oF
protocol datapath at the target (storage server) side, when the backend storage
devices are locally attached NVMe PCI devices.

After correctly setting up the offload and connections to clients, every
read/write/flush operation may be completely processed in the HCA at the target
side. No software will be running on the CPU to process those offloaded IO
operations. The HCA is utilizing the PCIe peer-to-peer capability to "talk"
directly to the NVMe drives over PCI. It is required that the system
architecture will allow such peer-to-peer communications.

The software at the server side is in charge of configuring the feature and
managing the NVMe-oF control communication with the clients (via NVMe-oF admin
queue). As response to these communications, connected QPs are created with each
client (by means of RDMA-CM, as defined in the NVMe-oF standard), which
represent NVMe-oF SQ and CQ pairs. Once a connection is created, the QP is
handed to the device to start offloading all IO commands.

Software is also required to handle any error cases, and IO commands that were
not configured to be offloaded.

NVMe-oF target offload datapath

Once properly configured and connections are established, the HCA will:

• Parse the RECVed NVMe-oF command capsule, and understand whether it is a READ
  / WRITE / FLUSH operation that should be offloaded.
• If this is a WRITE, the HCA will RDMA_READ the data from the client to local
  memory (unless it was inline with the command).
• The HCA will strip the NVMe command from the capsule, place it in an NVMe
  submit queue, and write to the submit queue doorbell.
• The HCA will poll the NVMe completion queue, and write to the completion queue
  doorbell.
• If this is a READ, the HCA will RDMA_WRITE the data from the local memory to
  the client.
• The HCA will SEND the the NVMe completion in a response capsule back to the
  client.

NVMe-oF target offload configuration

Setting up NVMe-oF target offload requires few steps:

1. Identify NVMe-oF offload capabilities of the device
2. Creating a SRQ with NVMe-oF offload attributes, to represent a single NVMe-oF
   subsystem
3. Creating NVMe backend device objects to represent locally attached NVMe
   subsystems
4. Setting up mappings between front-end facing namespace ids to a specific
   backend NVMe objects and namespace ids
5. Creating QPs connected with clients (using RDMA-CM, not in the scope of this
   document), bound to an SRQ with NVMe-oF offload
6. Modifying QP to enable NVMe-oF offload

Identify NVMe-oF offload capabilities

Software should call ibv_query_device_ex(), and test the returned
ibv_device_attr_ex.comp_mask for the avilability of NVMe-oF offload. If
available, the ibv_device_attr_ex.nvmf_caps struct holds the exact offload
capabilities and parameters of the device. These should be considered later
during the configuration.

Creating a SRQ with NVMe-oF offload attributes

A SRQ with NVMe-oF target offload represents a single NVMe-oF subsystem (a
storage target) to the fabric. Software should call ibv_create_srq_ex(), with
ibv_srq_init_attr_ex.srq_type set to nvmf_target_offload and
ibv_srq_init_attr_ex.nvmf_attr set to the specific offload parameters
requested for this SRQ. Parameters should be within the boundaries of the
respected capabilities. Along with the parameters, a staging buffer is provided
for the device use during the offload. This is a piece of memory allocated,
registered and provided via {mr, addr, len}. Software should not modify this
memory after creating the SRQ, as the device manages it by itself, and stores
there data that is in transit between network and NVMe device.

Note that this SRQ still has a receive queue. HCA will deliver to software
received commands that are not offloaded, and received commands from QPs
attached to the SRQ that are not configured with NVMF_OFFLOAD enabled.

Creating NVMe backend device objects

For the SRQ with NVMe-oF target offload feature to be able to submit work to
attached NVMe devices, software must provide the details of where to find the
NVMe submit queue, completion queue and their respective doorbells. How these
NVMe SQ, CQ and DBs are created is out of scope for this document. Normally
there should be an NVMe driver that owns the NVMe Admin Queue. By submitting
commands to this Admin Queue, SQs, CQs and DBs are generated. Software should
call ibv_srq_create_nvme_ctrl() with a set of NVMe {SQ, CQ, SQDB, CQDB} to
create an ibv_nvme_ctrl instance representing a specific NVMe backend
controller. These {SQ, CQ, SQDB, CQDB} should have been created exclusively for
this NVMe backend controller object, and this NVMe backend controller can be
used exclusively with the SRQ it was created for. SQ, CQ, SQDB and CQDB are all
provided by means of MR, address and possibly length (doorbells don't need
length as they have fixed 32 bit size). This means that those structures need to
be registered using ibv_reg_mr() before ibv_nvme_ctrl can be created.

Additionally, SQDB and CQDB initial values are provided.

Having NVMe objects created on SRQ does not yet allow servicing NVMe-oF to
clients. Namespace mappings that use these NVMe objects must be added.

Setting up namespaces mappings

When a client connects to an NVMe-oF subsystem, it will ask for the namespaces
list on that subsystem. Each namespace is identified by a namespace id (nsid),
that is then part of every IO request. The SRQ with NVMe-oF target offload
feature enabled will look at this nsid and map it to a specific nsid in one of
the NVMe backend objects created with it. Software should call
ibv_map_nvmf_nsid() to add such mappings to a SRQ. Each mapping consists of a
fabric-facing nsid and a set of {nvme_ctrl, nvme_nsid}. So IO operations
arriving from network for nsid will be submitted to nvme_ctrl, with a different
nvme_nsid. Software may create as many front-facing namespaces as needed, and
map them to different namespaces within the same nvme_ctrl or to namespaces in
different nvme_ctrls. However, as noted before, an nvme_ctrl may only be used in
mappings for the same SRQ it was created for.

After adding at least one namespace mapping, the SRQ as NVMe-oF target subsystem
is ready to service IOs.

Creating QPs

This stage is not different than any other normal QP creation and association
with SRQ. The NVMe-oF protocol standard requires that the first command on a QP
(that represents NVMe SQ) will be the CONNECT command capsule, and any other
commands should be responded with error. To meet the standard, the software
should not enable the QP NVme-oF offload (see next section) until after seeing
the CONNECT command. If a command different than CONNECT is received, software
should respond with error.

Modifying QP to enable NVMe-oF offload

Once a CONNECT command was received, software can modify the QP to enable its
NVMe-oF offload. ibv_modify_qp_nvmf() should be used to enable NVMe-oF
offload. From this point on, the HCA is taking ownership over the QP,
inspecting each command capsule received by the SRQ, and if this should be an
offloaded command, the flow described above is followed.

Note that enabling the NVMe-oF offload on the QP when created exposes the
solution to possible standard violation: if an IO command capsule will arrive
before the CONNECT request, the device will service it.

Errors and exceptions

Software should properly handle the following errors and exceptions:

1. Handle a non-offloaded IO request
2. Handle async events with QP type
3. Handle async events with SRQ type

Handle a non-offloaded IO request

This should be considered as a normal exception in case the SRQ was configured
to offload only part of the IO requests. In this case, software will receive
the completion on the CQ assigned with the QP, with the request in the SRQ.
Software should process the request, and it is allowed to generate RDMA
operations (reads, writes, sends) on the relevant QP in order to properly
terminate the transaction.

Handle async events with QP type

Software should listen to async events using ibv_get_async_event(). In case of
unrecoverable transport error hapenning on one of the offloaded QPs it will move
to error state and flush its queue. Since in normal operation the software may
not post to such QP and expect completions on it, the HCA will report an async
event indicating this QP has moved to error state. Software should treat this
as any other QP in error, i.e. close the connection and all its resources.

Handle async events with NVME_CTRL type

In case of unrecoverable error in HCA communication with an NVMe device, HCA
will report an async event indicating an error with NVME_CTRL. Software is
expected to remove this NVMe object and its related mappings.

[sashakot]

Add Mellanox's copyright in the top of the file

[sashakot]

Should we add a special function to "transport" that checks if the transport supports offload ?

[sashakot]

rc value from the previous call should be checked before the call

[sashakot]

We need to check that "offloaded" subsystem has only single name space

[mike-dubman]

@yshestakov , @sashakot - when do you plan to connect it to CI

[sashakot]

CI is not ready yet. @yshestakov will update when it's ready