Skip to content

OpenMPDK/dss-ecosystem

Repository files navigation

dss-ecosystem

DSS Ecosystem software consists of several software packages to interact with and benchmark DSS Object storage.

DSS Client Library

A customized S3 client library based on AWS SDK for C++ to support multiple logical clusters. DSS Client Library hides the underlying buckets from the user, allowing access to objects using corresponding keys.

DSS Client Library README

DSS Datamover

A horizontally scalable distributed command line tool to interact with DSS Object storage, supporting the following operations:

  • PUT: Uploaded file systems data to DSS object storage
    • Upload files from several client nodes and aggregate the upload status.
  • LIST: List the object keys from DSS object storage
    • Supports parallel listing of object keys from a single node
    • Supports distributed listing of object keys and aggregation of keys.
  • GET: Download objects from DSS object storage
    • Download objects to a shared file space
  • DELETE: Remove objects from DSS object storage

Additionally, there is support for RESUME operation to upload additional files excluded from a previous PUT.

DSS Datamover README

Benchmark Tools

The following are custom benchmark tools developed to measure performance of DSS Object storage.

DSS AI-Benchmark

A benchmark tool developed based on machine leaning (ML) framework such as PyTorch and TensorFlow to evaluate the read access performance of the DSS Object storage through ML model training.

DSS AI_benhcmark README

DSS s3-benchmark

A customized benchmark tool based on the original Wasabi s3-benchmark tool.

DSS s3-benchmark README