Flist format rewrite

[muhamadazmy]

Write now the flist format was optimized for size of the flist, by using capnp encoding format but this caused the following:

• Implementation of the fuse layer itself became very inefficient because of all the heavy lifting that need to be done to traverse the file tree
• Hard to modify flist after creation

But since we already using sqlite to store the data, we don't need the extra encoding layer from capanp instead we can normalize the inode information into the db in tables format. this will make querying, and traversing the filesystem much much faster and easier. This will also make modification of the flist metadata straight forward and can be implemented by other software or user tools.

Basics

The flist mainly contain information about the fiesystem, but not the file data itself. So all data stored in a filesystem Inode structure (this include size, type, mode, creation time, etc...) is stored. Another table that include the file blob information (mainly hash and encryption key)

From experience with working with 0-fs it's obvious that the most efficient structure to work with a filesystem is to map the inode structure directly to your database

// basic inode data
table Inode {
  ino: u64 primary key auto increment,
  parent: u64, // parent inode  
  name: string,
  uid: u32,
  gid: u32,
  size: u64,
  mode: u32,
  rdev: u32, // special for block/char devices (mknode)
  creationTime: timestamp,
  modifictionTime: timestamp,
}
// table that hold special inode data for example, target of symlinks and this need
// to be only set for special types 
table Data {
  ino: u64 foreign key ,
  data: bytes
}

// data blocks for regular files. The order matters because that's the block order in the same file
table Blocks {
  ino: u64 
  hash: hash primary key
  key: encryption key
}

// a metadata table that should container general information about the flist
// for example: block size, version, description. Keys must be known, extra keys are allowed
table Meta {
   key: string,
   value: string ?
}

// for partitioning of data
table Routing {
  start: [2]byte
  end: [2]byte
  store: string // url to store, this can be zdb:// url but can be other stores beased on the 0-fs implementation
 }

// same range can appear multiple times for redundancy 

• Meta and Routing are small data structures that should kept in memory during the operation of the database
• Root directiony / always has id 1, ino is auto increment number. this way all files gets unique ino
• mode reflect actual inode mode so this include type and permission as defined by stat.st_mode
• At least one routing record must exist in the Routing table to cover range 0x00 to 0xFF if all data exists on same data store
• the filesystem need to be able to support different storage types, must support zdb for start of course.

[muhamadazmy]

rfs (and other tools) can now support creation of flists metadata. the tool then can also upload the chunks to the remote and distributed storage locations and build the routing table.

[muhamadazmy]

Suggestion to name the new 0fs echo system (including the flist format) mkondo which means stream in Swahili