Home

Overview

This project provides a CLI and RPC-based APIs for creating and managing NVMeoF gateways for Ceph RBD.

The CLI tool connects to the server and generates gRPC requests. The gateway server accepts these requests to configure the NVMeoF target with SPDK. The gateway server stores these requests in order to restore the state of the NVMeoF target following reboot or outage. When two or more gateways are designated as a group, the gateway servers share a state store. A successful request to one gateway propagates to the entire group such that each member presents an identical NVMeoF target.

OMAP State Store

Gateway servers use OMAP, a key-value database within a Ceph object, to store NVMeoF target states. OMAP offers better performance for small, frequent writes than a Ceph object's data field, which reads in the entire field for each write operation. With each target state change request, servers initiate an atomic compare-and-write operation on the OMAP. If the server's local version matches the OMAP version, the request and a version increment are written to the OMAP. If there is a version mismatch, the write fails.

Watch/Notify and Polling

Each gateway group shares an OMAP. Each gateway server within a group registers a watch with the Ceph object containing this OMAP and can send notifications to other watchers of the same object. The gateways send these notify signals to indicate updates to the NVMeoF target. Other gateways, after receiving this signal, apply necessary changes to match their local state to that of the OMAP. In some cases, this notify signal can fail or timeout. To help maintain consistent state within groups, gateways also periodically poll the OMAP for changes.