The Khepri database library

Khepri is a tree-like replicated on-disk database library for Erlang and Elixir.

The basics

Data are stored in a tree structure. Each node in the tree is referenced by its path from the root node. A path is a list of Erlang atoms and/or binaries. For ease of use, Unix-like path strings are accepted as well.

For consistency and replication and to manage data on disk, Khepri relies on Ra, an Erlang implementation of the Raft consensus algorithm. In Ra parlance, Khepri is a state machine in a Ra cluster.

Project maturity

Khepri is still under active development and should be considered Beta at this stage.

Known limitations

Khepri currently hosts the entire data set in memory as well as on disk, so there is a realistic limit to how large a data set can be stored in it.

For this reason and others, storing blobs of files in Khepri is not recommended. For that, use an external blob store.

Documentation

• A short tutorial in the Getting started section below
• Documentation and API reference

Getting started

Add as a dependency

Add Khepri as a dependency of your project:

Using Rebar:

%% In rebar.config
{deps, [{khepri, "0.16.0"}]}.

Using Erlang.mk:

# In your Makefile
DEPS += khepri
dep_khepri = hex 0.16.0

Using Mix:

# In mix.exs
defp deps do
  [
    {:khepri, "0.16.0"}
  ]
end

Start default Khepri store

To start the default store, use khepri:start/0:

khepri:start().

The default Khepri store uses the default Ra system. Data is stored in the configured default Ra system data directory, which is khepri#$NODENAME in the current working directory.

It is fine to get started and play with Khepri. However, it is recommended to configure your own Ra system and Ra cluster to select the directory where data is stored and to be able to have multiple Khepri database instances running on the same Erlang node.

Insert data

Here's how to insert a piece of data, say, an email address of Alice:

%% Using a native path:
ok = khepri:put([emails, <<"alice">>], "alice@example.org").

%% Using a Unix-like path string:
ok = khepri:put("/:emails/alice", "alice@example.org").

Read data back

To get Alice's email address back, query the same path:

{ok, "alice@example.org"} = khepri:get("/:emails/alice").

Delete data

To delete Alice's email address:

ok = khepri:delete("/:emails/alice").

The emails parent node was automatically created when the alice node was inserted earlier. It has no data attached to it. However, after the alice node is deleted, the emails node will stay around. It is possible to tell Khepri to automatically remove emails as soon as its last child node is deleted. Khepri supports many more conditions by the way.

Transactional Operations

It is also possible to perform transactional queries and updates using anonymous functions, similar to Mnesia:

%% This transaction checks the quantity of wood left and returns `true` or
%% `false` if we need to process a new order.
khepri:transaction(
    fun() ->
        case khepri_tx:get([stock, wood]) of
            {ok, Quantity} when Quantity >= 100 ->
                %% There is enough wood left.
                false;
            _ ->
                %% There is less than 100 pieces of wood, or there is none
                %% at all (the node does not exist in Khepri). We need to
                %% request a new order.
                ok = khepri_tx:put([order, wood], 1000),
                true
        end
    end).

In this example, the transaction returns a boolean indicating if orders are ready to be processed. It does not send a message to a process or write something on disk for instance.

Because of the nature of the Raft consensus algorithm, transactions are not allowed to have side effects or take non-deterministic inputs such as the node name or the current date & time.

Triggers

Khepri supports stored procedures and triggers. They allow to store code in the database itself and automatically execute it after some event occurs.

1. Store an anonymous function in the tree:

   StoredProcPath = [path, to, stored_procedure],
   
   Fun = fun(Props) ->
             #{path := Path,
               on_action := Action} = Props
         end,
   
   khepri:put(StoreId, StoredProcPath, Fun).

2. Register a trigger using an event filter:

   %% A path is automatically considered a tree event filter.
   EventFilter = [stock, wood, <<"oak">>],
   
   ok = khepri:register_trigger(
          StoreId,
          TriggerId,
          EventFilter,
          StoredProcPath).

In the example above, as soon as the [stock, wood, <<"oak">>] node is created, updated or deleted, the anonymous function will be executed.

The function is executed at least once on the Ra leader's Erlang node. It may be executed multiple times if the leader changes and thus should be idempotent.

Unlike transaction functions, stored procedures may have whatever side effects they want.

Migrating from Mnesia

To help you migrate an existing Mnesia database, you can use the khepri_mnesia_migration application. It can take care of:

• synchronizing the cluster membership and
• copying Mnesia tables to a Khepri store.

How to build

Build

rebar3 compile

Build documentation

rebar3 edoc

Test

rebar3 xref
rebar3 eunit
rebar3 proper
rebar3 ct --sname ct
rebar3 as test dialyzer

Copyright and License

© 2021-2024 Broadcom. All Rights Reserved. The term "Broadcom" refers to Broadcom Inc. and/or its subsidiaries.

This work is dual-licensed under the Apache License 2.0 and the Mozilla Public License 2.0. Users can choose any of these licenses according to their needs.

The logo (doc/khepri-logo.svg) and the favicon (doc/khepri-favicon.svg) are based on the following two resources:

• https://www.svgrepo.com/svg/55105/rabbit (license: CC0)
• https://www.svgrepo.com/svg/336625/database-point (license: MIT)

SPDX-License-Identifier: Apache-2.0 OR MPL-2.0