Libri: opinionated, high-performance decentralized storage suitable for health data
Drausin Wulsin
We describe Libri, a decentralized storage network suitable for heterogenous healthcare data. Libri does not depend on any blockchain or distributed consensus protocol. This simplifies its implementation and allows its performance to scale very well with horizontal cluster size. Libri documents are automatically (re-)replicated across network peers to ensure robustness to planned and unplanned peer outages. The Libri API exposes simple Put\Get endpoints and a streaming Subscribe endpoint for storage notifications across the entire network. We describe initial experiments measuring request load and cluster size on request latency, rate, and throughput. In particular, a modest 64-peer cluster with 1 CPU and 6 GB RAM per peer easily handles over a million documents uploaded per day with median Get and Put latencies of 6 ms and 22 ms, respectively. Request latency and throughput scale well with horizontal cluster size increasing from 8 to 64 peers. These initial results give us confidence that Libri can scale to efficiently handle production-level load for healthcare use cases.