FungiStore: version 1

[despiegk]

features

• export to fs
• import from fs (blobs to ZDB, meta to FungiList) with support for relative location e.g. /data/fs/1/etc in FL is /etc
• mount FL optional copy on write to other dir (for the changes)
• metadata where blobs are stored is in the FungiList (multiple blob sources can be attached to 1 FL)
• FungiList is based on sqlite and nicely normalized tables (easy to read/write from any language)
• caching of blobs in local filesystem and can be used for multiple FungiMounts
• populate 1 ZDB from other ZDB based on flist (pre-warm a ZDB), do in X co-routines for more speed
• compress/encrypt blobs (encryption based on the hash)

Concepts

• FungiMount = mounted FL to a dir
• FungiList = FL = the list of metadata
• ZDB = backend where blobs are stored
• ZDB Cluster = group of ZDB's in 1 cluster so we have redundancy for ZDB storage

[muhamadazmy]

Changes to (rfs) (the 0-fs rust implementation) are here (PR is still open). please read the readme and the specs and let me know if u have any comment

branch v2-meta-rewrite

https://github.com/threefoldtech/rfs/tree/v2-meta-rewrite

[muhamadazmy]

https://gist.github.com/muhamadazmy/adf46289173ab65ca509c65f21b02f1e

[despiegk]

kristof will test this week

[muhamadazmy]

update:

From my side, fungistore rfs implementation is complete. The command line tool rfs supports creation, and mounting of .fl format, and upload blobs to different store types. Also ZOS now supports the new format as well (beside the hub) and has been tested to run workloads from user "custom" stores.

The only missing part of this story now is the section described under Fungi Cluster. Should we start workign on that?