Shelving uses a two part API - the consumer API is intended to serve the needs of users who want to read and write data. The implementer's API contains the raw machinery required to build a shelving store.
Shelving's fundamental model of a storage layer is that of a key value store. k/v stores are a simple abstraction, for which a large number of high quality implementations exist for instance redis, rocksdb and a host of others. k/v stores are especially attractive in the context of a project like Shelving, because most values are expected to be write once leaving great opportunity for replication and caching rather than having to design for random access writes which requiring central coordination.
At any point in time, a Shelving storage layer is expected to present three distinct "kinds" of information:
-
The current schema for the store, in the form of
clojure.specspecs as deserializing clients will understand it. This value will be requested (and checked) by connecting clients so they can verify their own integrity, and may be altered to achieve strict extension of the data model -
The vales in the store. Values are identified by a spec and a UUID. Compound values which have a
s/keysspec are expected to completely decompose and must be reconstituted from their relations. Other values such as strings and integers are expected to serialize and deserialize in the typical key/value pattern. -
The relations in the store. Several models are used for this, none of which need reflect the implementation's machinery. Relations are exposed as a sequence of
from toUUID pairs, identified by a[from to]relation pair. Primarily are consumed as a sequence of "to" UUIDs given a[from to]relation and an id.
No fundamental connection behavior is specified or expected.
open exists to differentiate between a connection's description and an active connection.
close is expected to close an open connection, freeing any related resources on both the client and any implementation or server.
(open config)
Opens a shelf for reading or writing.
Shelves must implement this method.
(flush conn)
Flushes (commits) an open shelf.
Shelves must implement this method.
By default throws me.arrdem.UnimplementedOperationException.
(close conn)
Closes an open shelf.
Shelves may implement this method.
By default just flushes.
The database schema describes what kinds of values it expects to store.
Clients MAY NOT write to relations or specs which have not been entered into the schema.
#'shelving.core/put-spec, the intentional write API, has support for automatically manipulating and extending the store's schema.
(schema conn)
Returns the schema record for a given connection.
Schemas are fixed when the connection is opened.
Shelves must implement this method.
By default throws me.arrdem.UnimplementedOperationException.
(set-schema conn schema)
Attempts to alter the live schema of the connection by applying the given transformer function to the current schema state with any additional arguments.
Implementation detail of null, which should be universally preferred. This method is an unprotected implementation detail not for general use.
Returns the schema record for a given connection.
Shelves must implement this method.
By default throws me.arrdem.UnimplementedOperationException.
These operations are used to manipulate specs, which approximate single data storage columns in a database. Specs may be written to, read from, scanned and counted. The counting operation is used to attempt to optimize query implementations.
Deletion of spec entries is not supported.
(enumerate-specs conn)
Enumerates all the known specs.
Shelves may provide alternate implementations of this method.
(enumerate-spec conn spec)
Enumerates all the known records of a spec by UUID.
Shelves must implement this method.
By default throws me.arrdem.UnimplementedOperationException.
(count-spec conn spec)
UNSTABLE: This API will probably change in the future
Returns an upper bound on the cardinality of a given spec.
The bound should be as tight as possible if not precise. Implementations of this method should be near constant time and should not require realizing the spec in question.
Shelves must implement this method.
By default throws me.arrdem.UnimplementedOperationException.
(put-spec conn spec id val)
The "raw" put operation on values. Inserts a fully decomposed value (tuple) into the designated spec, returning the ID at which it was inserted if an ID was not provided.
Users should universally prefer null. This method is an unprotected implementation detail not for general use.
Note that when inserting into "record" specs, all relations to the updated "record" ID must be invalidated.
Shelves must implement this method.
By default throws me.arrdem.UnimplementedOperationException.
(get-spec conn spec record-id)(get-spec conn spec record-id not-found)
Fetches a single tuple, being part of a record, from a shelf by its spec and ID.
Returns the record if it exists, otherwise returning the user-provided not-found value, taken to be nil by default.
Implementation detail of null, which should be preferred by users. This method is an unprotected implementation detail not for general use.
Shelves must implement this method.
By default throws me.arrdem.UnimplementedOperationException.
(has? conn spec record-id)
Indicates whether a shelf has a record of a spec.
Returns true if and only if the shelf contains a record if the given spec and ID. Otherwise must return false.
Implementations may provide alternate implementations of this method.
Relations well relate the IDs of values of one spec to the IDs of values of another spec. Like the values themselves, the relations can be read from, written to and counted.
Deletion of relations is not supported.
(enumerate-rels conn)
Enumerates all the known rels by ID (their [from-spec to-spec] pair). Includes aliases.
Shelves may provide alternate implementation of this method.
(enumerate-rel conn rel-id)
Enumerates the (from-id to-id) pairs of the given rel(ation).
Shelves must implement this method.
By default throws me.arrdem.UnimplementedOperationException.
(count-rel conn rel-id)
UNSTABLE: This API will probably change in the future
Returns an upper bound on the cardinality of a given relation.
The bound should be as tight as possible if not precise. Implementations of this method should be near constant time and should not require realizing the rel in question.
Shelves must implement this method.
By default throws me.arrdem.UnimplementedOperationException.
(put-rel conn spec rel-id from-id to-id)
The "raw" put operation on relations.
Inserts a [from rel to] triple into the data store unconditionally.
Users should universally prefer null. This method is an unprotected implementation detail not for general use.
Shelves must implement this method.
By default throws me.arrdem.UnimplementedOperationException.
(get-rel conn rel-id spec id)
UNSTABLE: This API will probably change in the future
Given a rel(ation) and the ID of an record of the from-rel spec, return a seq of the IDs of records it relates to. If the given ID does not exist on the left side of the given relation, an empty seq must be produced.
If the given ID does not exist on the left side of the given relation, an empty seq must be produced.
Note that if the rel [a b] was created with null, the rel [b a] also exists and is the complement of mapping from as to bs defined by [a b].
By default uses #'shelving.impl/enumerate-rel to do a full scan of the pairs constituting this relation.
Shelves may provide more efficient implementations of this method.