MultiKV is a simple and extensible library to manage file and path-based key/value stores for multiple storage backends, such as local storage, AWS S3 and Google Cloud Storage.
This library is not intended to be high performance, support high volume workloads or store large amounts of data. Instead, the goal is to provide a quick and easy way to create simple key/value stores. Additional features and optimizations such as encryption at rest, replication, authentication, authorization, and others are left to be managed by the storage backends (e.g. KMS encryption in a S3 or GCS bucket).
go get github.com/marcelocarlos/multikv
Initialize the local backend, then write and read from the KV store:
package main
import (
"fmt"
"github.com/marcelocarlos/multikv"
"github.com/marcelocarlos/multikv/backends/local"
)
func main() {
// Initializing backend
backend, err := local.NewLocalBackend("/tmp/multikv")
if err != nil {
fmt.Println(err)
}
// Initializing kv
kv := multikv.KV{Backend: backend}
// Put example
err = kv.Put("test/key", []byte("test"))
if err != nil {
fmt.Println(err)
}
// Get example
val, err := kv.Get("test/key")
if err != nil {
fmt.Println(err)
}
fmt.Println(string(val))
}
Tests are executed in CI, but if you want to run them locally first, run:
# Run lint
docker run --rm -v $(pwd):/app -w /app golangci/golangci-lint:v1.41.0 golangci-lint run
# Test - we use fake-gcs-server to spin up a local GCS server so we can safely run the tests
docker run --rm -d --name fake-gcs-server -p 4443:4443 fsouza/fake-gcs-server -public-host localhost:4443
go test -v ./...
The local
backend allows multikv
to use a local filesystem as the key/value storage layer. See the example usage we provided earlier see it in practice.
The gcs
backend allows multikv
to use Google Cloud Storage (GCS) as the key/value storage layer. To use it, you can tweak the example usage we provided earlier and change the backend, for example:
import (
// ...
"cloud.google.com/go/storage"
"github.com/marcelocarlos/multikv/backends/gcs"
// ...
)
func main() {
// ...
ctx := context.Background()
client, err := storage.NewClient(ctx)
backend := gcs.NewGCSBackend(client, "my-gcs-bucket", ctx)
// Initializing kv
kv := multikv.KV{Backend: backend}
// ...
}
Regardless the backend, each key/value pair generates 2 types files: data
and info
.
The data
file contains the base64-encoded value of the corresponding key
. The info
file contains JSON-encoded metadata about the key
using the following format:
{
"formatVersion": "1.0",
"kind": "info",
"path": "/path/to/my/key",
"createdAt": "2021-04-23T18:25:43.511Z",
"updatedAt": "2021-04-23T18:26:12.312Z"
}
There is no fixed roadmap yet, but planned features include:
- Additional backends
- S3
- Google Drive (via API, you can already use it via the local backend)
- Dropbox (via API, you can already use it via the local backend)
- Backblaze B2 backend
- KV features
- PBE encryption (client-side encryption)
- value versioning support
- value versioning optimizations (e.g. store diffs instead of full data for each version)