This library implements content-addressable storage types and protocols for Clojure. Content-addressable storage has several useful properties:
- Data references are abstracted away from the knowledge of where and how the blocks are stored, and so can never be 'stale'.
- Blocks are immutable, so there's no concern over having the 'latest version' of something - you either have it, or you don't.
- References are secure, because a client can re-compute the digest to ensure they have received the original data unaltered.
- Synchronizing data between stores only requires enumerating the stored blocks in each and exchanging missing ones.
- Data can be structurally shared by different higher-level constructs. For example, a file's contents can be referenced by different versions of metadata without duplicating the file data.
Library releases are published on Clojars. To use the latest version with Leiningen, add the following dependency to your project definition:
A block is a sequence of bytes identified by the cryptographic digest of its
content. All blocks have an :id and a :size - the block identifier is a
multihash value, and the size is the
number of bytes in the block content. Blocks may also have a :stored-at
value, which is the instant the backing store received the block.
=> (require '[blocks.core :as block])
;; Read a block into memory:
=> (def hello (block/read! "hello, blocks!"))
#'user/hello
=> hello
#blocks.data.Block
{:id #multi/hash "hash:sha2-256:d2eef339d508c69fb6e3e99c11c11fc4fc8c035d028973057980d41c7d162684",
:size 14,
:stored-at #inst "2019-02-18T07:02:28.751Z"}
=> (:id hello)
#multi/hash "hash:sha2-256:d2eef339d508c69fb6e3e99c11c11fc4fc8c035d028973057980d41c7d162684",
=> (:size hello)
14
;; Write a block to some output stream:
=> (let [baos (java.io.ByteArrayOutputStream.)]
(block/write! hello baos)
(String. (.toByteArray baos)))
"hello, blocks!"Internally, blocks either have a buffer holding the data in memory, or a reader which can be invoked to create new input streams for the block content. A block with in-memory content is a loaded block while a block with a reader is a lazy block.
=> (block/loaded? hello)
true
;; Create a block from a local file:
=> (def readme (block/from-file "README.md"))
#'user/readme
;; Block is lazily backed by the file on disk:
=> (block/loaded? readme)
false
=> (block/lazy? readme)
trueTo abstract over the loaded/lazy divide, you can create an input stream over a
block's content using open:
=> (slurp (block/open hello))
"hello, blocks!"
;; You can also provide a start/end index to get a range of bytes:
=> (with-open [content (block/open readme {:start 0, :end 32})]
(slurp content))
"Block Storage\n=============\n\n[!["A block's properties and content cannot be changed after construction, but blocks do support metadata. In order to guard against the content changing in the underlying storage layer, blocks can be validated by re-reading their content:
;; In-memory blocks will never change:
=> (block/validate! hello)
nil
;; But if the README file backing the second block is changed:
=> (block/validate! readme)
; IllegalStateException Block hash:sha2-256:515c169aa0d95... has mismatched content
; blocks.core/validate! (core.clj:115)
;; Metadata can be set and queried:
=> (meta (with-meta readme {:baz 123}))
{:baz 123}A block store is a system which saves and retrieves block data. Block stores have a very simple interface: they must store, retrieve, and enumerate the contained blocks. The simplest type of block storage is a memory store, which is backed by a map in memory. Another basic example is a store backed by a local filesystem, where blocks are stored as files in a directory.
The block storage protocol is comprised of five methods:
list- enumerate the stored blocks as a streamstat- get metadata about a stored blockget- retrieve a block from the storeput!- add a block to the storedelete!- remove a block from the store
These methods are asynchronous operations which return manifold deferred values. If you want to treat them synchronously, deref the responses immediately.
;; Create a new memory store:
=> (require 'blocks.store.memory)
=> (def store (block/->store "mem:-"))
#'user/store
=> store
#blocks.store.memory.MemoryBlockStore {:memory #<Ref@2573332e {}>}
;; Initially, the store is empty:
=> (block/list-seq store)
()
;; Lets put our blocks in the store so they don't get lost:
=> @(block/put! store hello)
#blocks.data.Block
{:id #multi/hash "hash:sha2-256:d2eef339d508c69fb6e3e99c11c11fc4fc8c035d028973057980d41c7d162684",
:size 14,
:stored-at #inst "2019-02-18T07:06:43.655Z"}
=> @(block/put! store readme)
#blocks.data.Block
{:id #multi/hash "hash:sha2-256:94d0eb8d13137ebced045b1e7ef48540af81b2abaf2cce34e924ce2cde7cfbaa",
:size 8597,
:stored-at #inst "2019-02-18T07:07:06.458Z"}
;; We can `stat` block ids to get metadata without content:
=> @(block/stat store (:id hello))
{:id #multi/hash "hash:sha2-256:94d0eb8d13137ebced045b1e7ef48540af81b2abaf2cce34e924ce2cde7cfbaa",
:size 14,
:stored-at #inst "2019-02-18T07:07:06.458Z"}
;; `list` returns the blocks, and has some basic filtering options:
=> (block/list-seq store :algorithm :sha2-256)
(#blocks.data.Block
{:id #multi/hash "hash:sha2-256:94d0eb8d13137ebced045b1e7ef48540af81b2abaf2cce34e924ce2cde7cfbaa",
:size 8597,
:stored-at #inst "2019-02-18T07:07:06.458Z"}
#blocks.data.Block
{:id #multi/hash "hash:sha2-256:d2eef339d508c69fb6e3e99c11c11fc4fc8c035d028973057980d41c7d162684",
:size 14,
:stored-at #inst "2019-02-18T07:06:43.655Z"})
;; Use `get` to fetch blocks from the store:
=> @(block/get store (:id readme))
#blocks.data.Block
{:id #multi/hash "hash:sha2-256:94d0eb8d13137ebced045b1e7ef48540af81b2abaf2cce34e924ce2cde7cfbaa",
:size 8597,
:stored-at #inst "2019-02-18T07:07:06.458Z"}
;; You can also store them directly from a byte source like a file:
=> @(block/store! store (io/file "project.clj"))
#blocks.data.Block
{:id #multi/hash "hash:sha2-256:95344c6acadde09ecc03a7899231001455690f620f31cf8d5bbe330dcda19594",
:size 2013,
:stored-at #inst "2019-02-18T07:11:12.879Z"}
=> (def project-hash (:id *1))
#'user/project-hash
;; Use `delete!` to remove blocks from a store:
=> @(block/delete! store project-hash)
true
;; Checking with stat reveals the block is gone:
=> @(block/stat store project-hash)
nil