Skip to content
This repository has been archived by the owner on Aug 28, 2021. It is now read-only.

Latest commit

 

History

History

nbs

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
benchmarks
benchmarks
 
 
frag
frag
 
 
test
test
 
 
NBS-on-AWS.md
NBS-on-AWS.md
 
 
README.md
README.md
 
 
aws_chunk_source.go
aws_chunk_source.go
 
 
aws_chunk_source_test.go
aws_chunk_source_test.go
 
 
aws_table_persister.go
aws_table_persister.go
 
 
aws_table_persister_test.go
aws_table_persister_test.go
 
 
block_store_test.go
block_store_test.go
 
 
cache.go
cache.go
 
 
conjoiner.go
conjoiner.go
 
 
conjoiner_test.go
conjoiner_test.go
 
 
dynamo_fake_test.go
dynamo_fake_test.go
 
 
dynamo_manifest.go
dynamo_manifest.go
 
 
dynamo_manifest_test.go
dynamo_manifest_test.go
 
 
dynamo_table_reader.go
dynamo_table_reader.go
 
 
dynamo_table_reader_test.go
dynamo_table_reader_test.go
 
 
factory.go
factory.go
 
 
factory_test.go
factory_test.go
 
 
fd_cache.go
fd_cache.go
 
 
fd_cache_test.go
fd_cache_test.go
 
 
file_manifest.go
file_manifest.go
 
 
file_manifest_test.go
file_manifest_test.go
 
 
file_table_persister.go
file_table_persister.go
 
 
file_table_persister_test.go
file_table_persister_test.go
 
 
fs_table_cache.go
fs_table_cache.go
 
 
fs_table_cache_test.go
fs_table_cache_test.go
 
 
manifest.go
manifest.go
 
 
manifest_cache.go
manifest_cache.go
 
 
manifest_cache_test.go
manifest_cache_test.go
 
 
mem_table.go
mem_table.go
 
 
mem_table_test.go
mem_table_test.go
 
 
mmap_table_reader.go
mmap_table_reader.go
 
 
mmap_table_reader_test.go
mmap_table_reader_test.go
 
 
persisting_chunk_source.go
persisting_chunk_source.go
 
 
persisting_chunk_source_test.go
persisting_chunk_source_test.go
 
 
root_tracker_test.go
root_tracker_test.go
 
 
s3_fake_test.go
s3_fake_test.go
 
 
s3_table_reader.go
s3_table_reader.go
 
 
s3_table_reader_test.go
s3_table_reader_test.go
 
 
stats.go
stats.go
 
 
stats_test.go
stats_test.go
 
 
store.go
store.go
 
 
table.go
table.go
 
 
table_persister.go
table_persister.go
 
 
table_persister_test.go
table_persister_test.go
 
 
table_reader.go
table_reader.go
 
 
table_set.go
table_set.go
 
 
table_set_test.go
table_set_test.go
 
 
table_test.go
table_test.go
 
 
table_writer.go
table_writer.go
 
 

README.md

Noms Block Store

A horizontally-scalable storage backend for Noms.

Overview

NBS is a storage layer optimized for the needs of the Noms database.

NBS can run in two configurations: either backed by local disk, or backed by Amazon AWS.

When backed by local disk, NBS is significantly faster than LevelDB for our workloads and supports full multiprocess concurrency.

When backed by AWS, NBS stores its data mainly in S3, along with a single DynamoDB item. This configuration makes Noms "effectively CA", in the sense that Noms is always consistent, and Noms+NBS is as available as DynamoDB and S3 are. This configuration also gives Noms the cost profile of S3 with power closer to that of a traditional database.

Details

  • NBS provides storage for a content-addressed DAG of nodes (with exactly one root), where each node is encoded as a sequence of bytes and addressed by a 20-byte hash of the byte-sequence.
  • There is no update or delete -- only insert, update root and garbage collect.
  • Insertion of any novel byte-sequence is durable only upon updating the root.
  • File-level multiprocess concurrency is supported, with optimistic locking for multiple writers.
  • Writers need not worry about re-writing duplicate chunks. NBS will efficiently detect and drop (most) duplicates.

Perf

For the file back-end, perf is substantially better than LevelDB mainly because LDB spends substantial IO with the goal of keeping KV pairs in key-order which doesn't benenfit Noms at all. NBS locates related chunks together and thus reading data from a NBS store can be done quite alot faster. As an example, storing & retrieving a 1.1GB MP4 video file on a MBP i5 2.9Ghz:

  • LDB
    • Initial import: 44 MB/s, size on disk: 1.1 GB.
    • Import exact same bytes: 35 MB/s, size on disk: 1.4 GB.
    • Export: 60 MB/s
  • NBS
    • Initial import: 72 MB/s, size on disk: 1.1 GB.
    • Import exact same bytes: 92 MB/s, size on disk: 1.1GB.
    • Export: 300 MB/s

Status

NBS is more-or-less "beta". There's still work we want to do, but it now works better than LevelDB for our purposes and so we have made it the default local backend for Noms:

# This uses nbs locally:
./csv-import foo.csv /Users/bob/csv-store::data

The AWS backend is available via the aws: scheme:

./csv-import foo.csv aws:table/bucket/database::data