Move storage traits implementation to the fuel-core-storage crate

[xgreenx]

Overview

Closes #1548
Closes #430

The change moves the implementation of the storage traits for required tables from fuel-core to fuel-core-storage crate. The change also adds a more flexible configuration of the encoding/decoding per the table and allows the implementation of specific behaviors for the table in a much easier way. It unifies the encoding between database, SMTs, and iteration, preventing mismatching bytes representation on the Rust type system level. Plus, it increases the re-usage of the code by applying the same structure to other tables.

It is a breaking PR because it changes database encoding/decoding for some tables.

StructuredStorage

The change adds a new type StructuredStorage. It is a wrapper around the key-value storage that implements the storage traits(StorageInspect, StorageMutate, StorageRead, etc) for the tables with structure. This structure works in tandem with the TableWithStructure trait. The table may implement TableWithStructure specifying the structure, as an example:

impl TableWithStructure for ContractsRawCode {
    type Structure = Plain<Raw, Raw>;

    fn column() -> Column {
        Column::ContractsRawCode
    }
}

It is a definition of the structure for the ContractsRawCode table. It has a plain structure meaning it simply encodes/decodes bytes and stores/loads them into/from the storage. As a key codec and value codec, it uses a Raw encoding/decoding that simplifies writing bytes and loads them back into the memory without applying any serialization or deserialization algorithm.

If the table implements TableWithStructure and the selected codec satisfies all structure requirements, the corresponding storage traits for that table are implemented on the StructuredStorage type.

Codecs

Each structure allows customizing the key and value codecs. It allows the use of different codecs for different tables, taking into account the complexity and weight of the data and providing a way of more optimal implementation.

That property may be very useful to perform migration in a more easier way. Plus, it also can be a no_std migration potentially allowing its fraud proving.

An example of migration:

/// Define the table for V1 value encoding/decoding.
impl TableWithStructure for ContractsRawCodeV1 {
    type Structure = Plain<Raw, Raw>;

    fn column() -> Column {
        Column::ContractsRawCode
    }
}

/// Define the table for V2 value encoding/decoding.
/// It uses `Postcard` codec for the value instead of `Raw` codec.
///
/// # Dev-note: The columns is the same.
impl TableWithStructure for ContractsRawCodeV2 {
    type Structure = Plain<Raw, Postcard>;

    fn column() -> Column {
        Column::ContractsRawCode
    }
}

fn migration(storage: &mut Database) {
    let mut iter = storage.iter_all::<ContractsRawCodeV1>(None);
    while let Ok((key, value)) = iter.next() {
        // Insert into the same table but with another codec.
        storage.storage::<ContractsRawCodeV2>().insert(key, value);
    }
}

Structures

The structure of the table defines its behavior. As an example, a Plain structure simply encodes/decodes bytes and stores/loads them into/from the storage. The SMT structure builds a sparse merkle tree on top of the key-value pairs.

Implementing a structure one time, we can apply it to any table satisfying the requirements of this structure. It increases the re-usage of the code and minimizes duplication.

It can be useful if we decide to create global roots for all required tables that are used in fraud proving.

impl TableWithStructure for SpentMessages {
    type Structure = Plain<Raw, Postcard>;

    fn column() -> Column {
        Column::SpentMessages
    }
}
                 |
                 |
                \|/

impl TableWithStructure for SpentMessages {
    type Structure =
        Sparse<Raw, Postcard, SpentMessagesMerkleMetadata, SpentMessagesMerkleNodes>;

    fn column() -> Column {
        Column::SpentMessages
    }
}

Side changes

iter_all

The iter_all functionality now accepts the table instead of K and V generics. It is done to use the correct codec during deserialization. Also, the table definition provides the column.

Duplicated unit tests

The fuel-core-storage crate provides macros that generate unit tests. Almost all tables had the same test like get, insert, remove, exist. All duplicated tests were moved to macros. The unique one still stays at the same place where it was before.

StorageBatchMutate

Added a new StorageBatchMutate trait that we can move to fuel-storage crate later. It allows batch operations on the storage. It may be more performant in some cases.

/// The traits allow work with the storage in batches.
/// Some implementations can perform batch operations faster than one by one.
pub trait StorageBatchMutate<Type: Mappable>: StorageMutate<Type> {
    /// Initialize the storage with batch insertion. This method is more performant than
    /// [`Self::insert_batch`] in some case.
    ///
    /// # Errors
    ///
    /// Returns an error if the storage is already initialized.
    fn init_storage(
        &mut self,
        set: &mut dyn Iterator<Item = (&Type::Key, &Type::Value)>,
    ) -> Result<()>;

    /// Inserts the key-value pair into the storage in batch.
    fn insert_batch(
        &mut self,
        set: &mut dyn Iterator<Item = (&Type::Key, &Type::Value)>,
    ) -> Result<()>;

    /// Removes the key-value pairs from the storage in batch.
    fn remove_batch(&mut self, set: &mut dyn Iterator<Item = &Type::Key>) -> Result<()>;
}

Follow-up

It is one of the changes in the direction of the forkless upgrades for state transition functions and fraud proofs. The idea behind this is that the fuel_core_executor::Executor will work directly with the StructuredStorage instead of the Database. It will perform only state transition-related modifications to the storage, while all outside modifications like updating of receipts, transition status, block insertions, messages removing, and transaction storing will be a part of another service/process.

[Voxelot]

Maybe we should use dedicated encoding errors instead of anyhow?

[xgreenx]

Maybe the problem is that I don't know what the enum will look like=D We have only two errors right now: postcard and TryFromSliceError.

The idea is to make the Decode trait as generic as possible and it means something like:

enum DecodeError {
    TryFromSliceError(TryFromSliceError),
    Other(anyhow::Error)
}

And I decided to go simply with anyhow::Error.

[Voxelot]

Suggested change

	pub struct KeyConvertor;
	pub struct KeyConverter;

[Voxelot]

Suggested change

	/// [`Self::insert_batch`] in some case.
	/// [`Self::insert_batch`] in some cases.

[Voxelot]

Would this not apply anymore once we separate executor logic from other kinds of data?

[xgreenx]

Yes, we will remove it after merging #1579 and resolving #1583 and #1580

[Voxelot]

Is this a check we should do at compile time (i.e. const eval) rather than every time a ForeignColumn is instantiated?

[xgreenx]

It is impossible to do during compile time because the id can be known only during execution. We could try to add support for const fn new, but it doesn't worth it. We don't use ForeignColumn for our logic. It exists for anyone who decides to implement something on top of us=)

But I think in the follow-up PRs(after extractions of off-chain logic from the executor), maybe we will even remove that variant.

[MitchTurner]

I'd prefer a generic/impl here over type erasure. Is there a specific reason we want dyn over having some concrete type at compile time?

[bvrooman]

Suggested change

	/// Teh tables allows to iterate over all transactions owned by an address.
	/// These tables allow iteration over all transactions owned by an address.

[bvrooman]

Suggested change

	use fuel_core_types::fuel_vm::ContractsAssetKey;
	use fuel_core_types::fuel_vm::{
	ContractsAssetKey,
	ContractsStateKey,
	};

[bvrooman]

Nit: Maybe use a consistent name for generics for Encode/Decode since K/V implies Key and Value

[bvrooman]

The type of value written to the database is () - Is this right?

[xgreenx]

Added additional test to prove all possible use cases