Skip to content

Blob stores abstractions. Supports AWS S3, Google Storage, Azure Blob File Storage, and local FS

License

Notifications You must be signed in to change notification settings

streamingfast/dstore

Repository files navigation

StreamingFast Storage Abstraction

reference License

dstore is a simple abstraction on top of Local storage and Cloud storage. It handles commonly used functions to store things (locally, or on cloud storage providers), list files, delete, etc..

It is used by StreamingFast.

Features

It currently supports:

  • AWS S3 (s3://[bucket]/path?region=us-east-1, with AWS-specific env vars)
    • Minio (through the S3 interface)
  • Google Storage (gs://[bucket]/path, with GOOGLE_APPLICATION_CREDENTIALS env var set)
  • Azure Blob Storage (az://[account].[container]/path, with AZURE_STORAGE_KEY env var set)
  • Local file systems (including virtual of fused-based) (file:/// prefix)

Testing

The storetests package contains all our integration tests we perform on our store implementation. Some of the store implementations can be tested directly while few others, from Cloud Providers essentially, requires some extra environment variables to run. They are skip if the correct environment variables for the provider is not set.

To run the full test suite, you will need to perform the following steps.

First, you will need to have locally a few dependencies:

Then, start minio server:

mkdir -p /tmp/minio-tests/store-tests
cd /tmp/minio-tests
minio server .

Ensure you have access to GCP Storage Bucket, S3 Bucket, then run the full test suite:

STORETESTS_GS_STORE_URL="gs://streamingfast-developement-random/store-tests"\
STORETESTS_S3_STORE_URL="s3://streamingfast-customer-outbox/store-tests?region=us-east-2"\
STORETESTS_S3_MINIO_STORE_URL="s3://localhost:9000/store-tests?region=none&insecure=true&access_key_id=minioadmin&secret_access_key=minioadmin"\
go test ./...

Note S3 bucket is not existing anymore, you need to provide your own.

Contributing

Issues and PR in this repo related strictly to the dstore library.

Report any protocol-specific issues in their respective repositories

Please first refer to the general StreamingFast contribution guide, if you wish to contribute to this code base.

License

Apache 2.0

About

Blob stores abstractions. Supports AWS S3, Google Storage, Azure Blob File Storage, and local FS

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages