Skip to content

everest-engineering/lhotse-storage

Repository files navigation

File storage support for Lhotse

Build status Quality Gate Status

This is a standalone library as well as a supporting repository for Lhotse, a starter kit for writing event sourced web applications following domain driven design principles.

Purpose

This library implements two file stores: one is referred to as permanent, the other as the ephemeral store. The permanent file store is for storing critical files that, such as user uploads, cannot be recovered. The ephemeral store is for non-critical files that can be regenerated by the system either dynamically or via an event replay.

Our file store implementation automatically deduplicates files. Storing a file whose contents matches a previous file will return a (new) file identifier mapping to the original. The most recently stored file will then be silently removed.

File stores need backing service such as a blob store or filesystem. This library supports an in-memory file store for testing and development, Mongo GridFS and AWS S3.

Configuring the In-Memory Filestore

The in-memory filestore backend is intended only for development and testing. This filestore is not distributed so will not work well when running multiple instances of the application in HA mode. A locally hosted or AWS hosted S3 compatible filestore is a better bet in this instance.

application.filestore.backend=inMemory

Configuring Mongo GridFS

Set the application property:

application.filestore.backend=mongoGridFs

Configuring AWS S3

Set the following application properties:

application.filestore.backend=awsS3
application.filestore.awsS3.buckets.permanent=sample-bucket-permanent
application.filestore.awsS3.buckets.ephemeral=sample-bucket-ephemeral

We rely on DefaultAWSCredentialsProviderChain and DefaultAwsRegionProviderChain for fetching AWS credentials and the AWS region.

License

License

License: EverestEngineering

Talk to us hi@everest.engineering.