Merkle Tree storage implementation

[magestrio]

Feature type

• UI
• Implementation
• Specification
• CI/CD
• Other

Feature description

Implemented a Poseidon Sparse Merkle Tree that supports storing it on IPFS in both JSON and IPDL formats. This approach is fully aligned with the Merkle Tree from the Snarkyjs lib.

Pull request checklist

Please check if your PR fulfills the following requirements:

• Your code builds clean without any errors or warnings.
• Tests for changes have been added (for bug fixes / features). If no, why?

[iam-robi]

if this is a test file , I think it's good to have a description of each test, and also more comments on each step. it would be extremely helpful to understand more quickly what you are trying to achieve

[iam-robi]

ideally also, we would be able to run tests with jest, and/or from a script in package.json this will be very helpful to run locally , understand the purpose of each component. some tests like : "checking structure of merkle tree" "storing parse merkle tree" , "storing basic merkle"

[JoE11-y]

All tests work locally provided a new Merkle tree instance is passed as an argument.

[JoE11-y]

Did some more tests and realized that when trying to use a witness to calculate the root of the Merkle tree, it returns the wrong root.
Is there any way to get the root using a witness?

Code Snippet

class MyMerkleWitness extends createExtendedMerkleWitness(8) { }

const Tree = new MerkleTreeJSON(8, storage);
//const Tree = new MerkleTree(8);

Tree.setLeaf(0n, Field(56));

// check
let w = await Tree.getWitness(0n);

let witness = new MyMerkleWitness(w);

let currentRoot = await Tree.getRoot();
console.log(currentRoot.toString())
console.log(witness.calculateRoot(Field(56)).toString())

issue resolved by adding the await to the setLeaf method
await Tree.setLeaf(0n, Field(56))

rookie mistake

[chiro-hiro]

I think, this implementation is a bit out-date. Please consider to rebase and use the new implementation of StorageEngineIPFS instead of old js-ipfs.

[chiro-hiro]

I would prefer, to use BSON instead of JSON since it's more friendly with binary and also the serialization/deserialization much more better than JSON.

[chiro-hiro]

Overall everything looks good to me, there is something we need to change due to the recent change to storage engine.

1. Im prefer to use StorageEngineIPFS in Feature/ipfs storage engine #9 instead of js-ipfs since it's deprecated soon.
2. Use BSON instead of JSON
3. We should use a pure poseidon Merkle tree instead of put CID in the Merkle tree. Since the data identify is its has not CID, handling CID and Merkle tree link is something that we do in the background.

[chiro-hiro]

Did some more tests and realized that when trying to use a witness to calculate the root of the Merkle tree, it returns the wrong root. Is there any way to get the root using a witness?

Code Snippet

class MyMerkleWitness extends createExtendedMerkleWitness(8) { }

const Tree = new MerkleTreeJSON(8, storage);
//const Tree = new MerkleTree(8);

Tree.setLeaf(0n, Field(56));

// check
let w = await Tree.getWitness(0n);

let witness = new MyMerkleWitness(w);

let currentRoot = await Tree.getRoot();
console.log(currentRoot.toString())
console.log(witness.calculateRoot(Field(56)).toString())

issue resolved by adding the await to the setLeaf method await Tree.setLeaf(0n, Field(56))

Did you mean, we're unable to create a valid witness for a given root digest?.

[JoE11-y]

Did some more tests and realized that when trying to use a witness to calculate the root of the Merkle tree, it returns the wrong root. Is there any way to get the root using a witness?

Code Snippet

class MyMerkleWitness extends createExtendedMerkleWitness(8) { }

const Tree = new MerkleTreeJSON(8, storage);
//const Tree = new MerkleTree(8);

Tree.setLeaf(0n, Field(56));

// check
let w = await Tree.getWitness(0n);

let witness = new MyMerkleWitness(w);

let currentRoot = await Tree.getRoot();
console.log(currentRoot.toString())
console.log(witness.calculateRoot(Field(56)).toString())

issue resolved by adding the await to the setLeaf method await Tree.setLeaf(0n, Field(56))

Did you mean, we're unable to create a valid witness for a given root digest?

At first, I did, but turns out I didn't wait for the promise so it returned the wrong root.
Everything works fine.

[chiro-hiro]

Btw we need to change the title of this PR as well, this PR is about Mekle tree construction/reconstruction and providing Merkle tree witness.

[chiro-hiro]

@magestrio Please update the code and resolve the conflict then ping me.