BlobModule API

[Wondertan]

// Blob represents any application-specific binary data that anyone can submit to Celestia.
type Blob struct {
	Namespace namespace.ID
	Data []byte
}

// Proof is a collection of nmt.Proofs that verifies the inclusion of the data.
type Proof []nmt.Proof

// Commitment is a Merkle Root of the subtree built from shares of the Blob.
type Commitment []byte

// Commitment computes the commitment of the Blob. 
// It is computed by splitting the blob into shares and building the Merkle subtree to be included after Submit.
func (b *Blob) Commitment() Commitment

// BlobModule provides all necessary
type BlobModule interface {
	// Submit sends PFB transaction and reports the height it landed on.
	// Allows sending multiple Blobs atomically synchronously.
	// Uses default wallet registered on the Node.
	Submit(context.Context, Blob...) (height uint64, error)
	
	// Get retrieves the blob by commitment. 
	Get(context.Context, uint64, namespace.ID, Commitment) (Blob, error)
	
	// GetAll retrieves all the blobs for given namespaces at the given height. 
	GetAll(context.Context, height uint64, namespace.ID...) ([]Blob, error)

    // GetProof gets the Proof for a blob if it exists at the given height under the namespace. 
    GetProof(context.Context, height uint64, namespace.ID, Commitment) (Proof, error)

	// Included checks whether a blob's given commitment(Merkle subtree root) is included at the given height under a particular namespace.
	// Useful for light clients wishing to check the inclusion of the data.
	Included(context.Context, height uint64, namespace.ID, Commitment, Proof) (bool, error)
}

Two uncovered/undecided features:

• Exposing inclusion proofs from the API. I am very hesitant to do this for reasons I will explain in a diff thread.

• Inclusion checks for data at a particular share index rather than the whole blob.

  cc @musalbas, @liamsi, @tzdybal, @nashqueue

[musalbas]

It might be helpful to add the index of the Blob when you get them, in case people need to link them to a PFB that was published (eg to link them to a certain transaction submitter).

[musalbas]

Ok, we can guarantee 1:1 parity between the order inside the data root and the slice returned by Get. This is what I meant with

That still doesn't allow you to know the index of the blob inside the data root though, as you don't know the index of the first share in the slice inside the data root.

And I assume users cannot select the index when the Submit PFB, right?

No, that's determined by block producers.

[preston-evans98]

Being able to easily identify the blob's location inside of the data root would simplify our node implementation, but I don't think it changes the amount of work we have to do in-circuit - so it's not a huge deal either way.

[preston-evans98]

To expand on this a little - IIRC, our current implementation just reconstructs the entire data square outside of the circuit. Since we already have the complete data, it's easy for us to find the index of a particular blob in the data root. And the verifier implicitly checks that index by verifying the merkle proof of inclusion.

But now that I think about it, isn't the index already included in the merkle proof?

[musalbas]

But now that I think about it, isn't the index already included in the merkle proof?

That should be true, yes, if the API includes proofs.

[mztacat]

indexing the BlobModule would be great. i find it really hard navigating here

[Wondertan]

Also, there is one feature request from rollkit that I want to understand fully: the need to manage blob inclusion on the rollkit side rather than delegating it to celestia-node. By manage, I mean handling networking and verifying them. On the other side, I am proposing to hide/abstract away most of the proof-related logic as an implementation detail of Celestia Node, as we do for DAHeader syncing and sampling.
The API looks like #1829 (comment)

Pros:

• Less work for Rollkit and other teams. Fetching and proof verification are done by cel-node
• Code for this is implemented in a single place, enabling better maintainability and easier audits. Having this all in an open place benefits our ecosystem.
• We can enable giving proofs in special cases, but the default is celestia-node handling this.
• Enables minimal future and fool proof API for Blobs.
  • Future proof because abstracts away proving system enabling this API to survive changes once Celestia migrates to KZG(not a fact)
  • Fool proof because it hided away complexity
• It does not increase development on Celestia-node team, we anyway have to have such features.

Cons:

• Technically increases networking load on celestia-node p2p network from rollkits, but
  • It's only proofs, not the data
  • In any case, it is accessible to group nodes interested in a particular namespace, so in practice, these proofs can be offloaded from cel-nodes hosted by celestia to cel-node hosted by rollkit network by discovery.
• Anything else you have in mind that I missed

[musalbas]

Regardless of how much is internalized in celestia-node, zk rollups will still always need to verify the NMT proofs in-circuit, so celestia-node will still need to expose the raw proofs so that they can be passed into the zk program, in any short- or long- term proposal.

"implement a custom and single-purpose solution on the Rollkit side" does not seem to meet that goal, since that may be fine for Rollkit, but it's not fine for other rollup frameworks such as https://github.com/Sovereign-Labs/Jupiter?

[Wondertan]

Celestia node, anyway, must be able to serve proofs, even when Rollkit handles networking independently because the sequencer has to get it from somewhere to share with his network. Jupiter will use the same way to request proofs; it just won't be as efficient as the custom networking that Rollkit aims for now.

[Wondertan]

The other question I have to Soverign design. Do they need blob inclusion proofs specifically, e.g spanned over multiple rows, or proofs over just rows is enough for them?

[preston-evans98]

Do you need to access the proof only after submission or also during the inclusion check? Do you need proofs outside of submission and inclusion check operations?

If I understand your question correctly, we just needs proofs during the inclusion check.

To be more explicit, we make a SNARK proof for each Celestia block that says "I have read all of the blobs from Namespace X". In order to do that, we have the prover download all of the blobs (for the namespace) from Celestia Node and provide them as inputs to the circuit. Then, the circuit checks an NMT proof that (1) all of the blobs "real" (meaning that they are committed to in the DAH) and (2) the set of blobs is "complete" (meaning that the prover didn't leave out any blobs).

[preston-evans98]

The other question I have to Soverign design. Do they need blob inclusion proofs specifically, e.g spanned over multiple rows, or proofs over just rows is enough for them?

We need to verify statements about blobs specifically

[Wondertan]

The alternative API which forces users to handle Proofs:

(only affected methods are shown)

	// Awaits inclusion and provides users proofs for the user(Rollkit) to gossip over their network
	Submit(context.Context, Blob...) (height uint64, []Proof, error)
	
	// Only find the correct DAHeader by height and perform an inclusion check against it(usefull for Rollkit Light clients)
	Available(context.Context, height uint64, namespace.ID, hash.Hash, Proof) (bool, error)

[liamsi]

API looks like a great start to me!

// Commit computes the hash commitment of the Blob.
func (b *Blob) Commit() hash.Hash

// Available checks whether the given hash commitment of a blob is available/included at the given height under a particular namespace.
// Useful for light clients wishing to check the inclusion of the data.
Available(context.Context, height uint64, namespace.ID, hash.Hash) (bool, error)

I think in both cases it remains a bit too vague what commit refers to. Is it the hash of the blob (probs not) is it the same substree root included in the pfb?

[Wondertan]

It's a non-interactive subtree root, yes. Will clarify that

[Wondertan]

Clarified

[rootulp]

// Commit computes hash commitment of the Blob.
func (b *Blob) Commit() []byte

Proposal to use commitment for consistency celestiaorg/celestia-app#1202.

is it the same substree root included in the pfb?

If this commitment refers to the ShareCommitments that a user generates and includes in their PFB transaction then it is not a singular subtree root. Instead it is a Merkle root over multiple subtree roots in the data square. Ref:

1. https://github.com/celestiaorg/celestia-app/tree/main/x/blob#how-is-the-sharecommitment-generated
2. https://github.com/rootulp/celestia-app/blob/d2eb6b0fd9f2b83859499ac315962ea7a67bd13f/x/blob/types/payforblob.go#L127-L133

[nashqueue]

Rollkit will need an inclusion check of arbitrary shares that might not convert into a commitment. Let me give you the example:

[S1][S2][S3][S4][S5][S6][S7][S8] This Blob is 8 Shares long and and you can generate a PFB commitment over it with ease.

Now lets say a full node found fraud in those shares and wants to share it with the LN. They will need to share a blob inclusion proof to the shares those specific shares.

Those shares might be in the middle of the block. Because of non-imnteractive default rules you cannot have a pfb commitment over those.

Lets say the fraudulent shares are in [S2][S3]

The merkle tree over the shares looks like this:
/ \ / \ / \ / \
[S1][S2][S3][S4][S5][S6][S7][S8]

Thats why there is no PFB commitment over
\ /
[S2][S3]

One way to reference those 2 Shares is by (height,index,length) and not by commitment.

[nashqueue]

S1-S8 is one Block. The compact fraud proof would only require the LN to get a small portion of the block. You only know if the fraud proof is invalid after you check the last ISR (in the fraud-proof). This means that you have to execute all Txs to be able to know if the last ISR is correct or not.

[rootulp]

This means that you have to execute all Txs to be able to know if the last ISR is correct or not.

In this context I think "you" is referring to the LN. If so, I don't see why this is problematic unless there is an assumption that the LN doesn't have hardware resource requirements to download S1 through S8. Is that the case?

One way to reference those 2 Shares is by (height,index,length) and not by commitment.

An alternative may be to provide an NMT inclusion proof to the subtree root that contain the shares with fraud. In the example I think a FN could convince a LN that S1-S4 exist in the NMT inner node b. The LN would have to download the shares S1-S4. Then I think other NMT proofs can be provided (with hashes and not the raw shares) to prove that S1-S4 is part of the blob S1-S8 and that this blob exists in the data square.

                a
              /   \
            /       \
          /           \
        b              c
      /   \         /     \
    d      e       f       g
  /  \    /  \   /   \   /   \
[S1][S2][S3][S4][S5][S6][S7][S8]

[nashqueue]

One of the verification steps of compact fraud proofs will be to see if the shares from the fraud proof exist in the blob that is referenced. And before that we presumably already verified the inclusion of the blob.

The reason why you want to have compact FPs instead of full blocks is still the fact that a LN who is processing a FP for every block for the whole block will turn into a FN. Maybe that is feasible resources wise, but that is not something desirable.

[rootulp]

is still the fact that a LN who is processing a FP for every block for the whole block will turn into a FN

Sorry for repeating myself but a LN that processes a valid FP for the first block will shut down. A LN that processes an invalid FP for the first block will block the FN that sent that FP.

I don't see why FPs need to reference a subset of shares rather than the entire blob. Given this API is for the blob module, I'm hesitant to expose the raw share type (and associated fraud proofs for them) from this API. If there is a convincing use case that warrants a need to index directly to shares, we may consider a separate share API for it.

[musalbas]

It's because there is indeed "an assumption that the LN doesn't have hardware resource requirements to download S1 through S8".

Compact fraud proofs are just an optimization for DoS-resistance to prevent attackers from making LNs download too much junk data (i.e. many multi-megabyte invalid fraud proofs coming from different IP addresses).

[evan-forbes]

We also use the share version, which is added to the share and is needed to properly recreate any commitments. Currently, we only have a share version of 0, but we might want to consider adding it here given its importance.

ref share specs

[rootulp]

Good question. I think user should have the option to specify namespaceVersion and shareVersion from the outset so that our APIs are capable of introducing new versions for these fields without a breaking change to the API.

If either of those fields aren't specified by the user, we can default to 0 b/c they will be the only supported namespaceVersion and shareVersion at mainnet launch.

[Wondertan]

Ok, to support the namespace version, we may do an additional namespace id type. This type will have versioning exposed, and users will be able to instantiate a namespace with the desired version. The namespace id type will then be used by our APIs.

For share versioning, it might be a bit trickier, as the blob module does not know about shares, but if it's really necessary we can have a share version on the Blob struct that signals to the node which version to use.

[Wondertan]

0 b/c they will be the only supported namespaceVersion and shareVersion at main net launch.

Still, I would like to have a future-proof version of the API. But the question is, once we introduce version e.g. 1, will there be any use case for users to use the previous version? I presume yes, but it could be that we only support versions to be able to parse older blocks as newer ones always use the new version.

[rootulp]

will there be any use case for users to use the previous version?

Yes because a new namespace version will be interpreted as a completely distinct namespace. In other words:

namespaceVersion := 0
namespaceId := []byte{1, 1, 1, ...} // 32 bytes

namespaceVersion := 1
namespaceId := []byte{1, 1, 1, ...} // 32 bytes

will publish to two distinct namespaces so roll-ups must query by namespaceVersion + namespaceId. In order to avoid breaking roll-ups on mainnet, celestia-node can't assume which version they want to use after we introduce more than one. In fact, it's likely safest to retain the default of namespaceVersion=0 after we introduce namespaceVersion 1 so that roll-ups must explicitly opt in to the new namespace version and therefore accept that they are migrating their roll-up's canonical namespace.

that we only support versions to be able to parse older blocks as newer ones always use the new version.

If we disabled ability to use previous namespace versions, that would be a breaking change and at the same time we could make a breaking change to this API.

[Wondertan]

And I guess the same is true for the share versioning, correct?

[nashqueue]

Should 'Submit' receive [ ] of Blobs? If not how do you submit multiple blobs under the same PFB?

With this Interface, we would do this:

• Create Commitment of Data
• calculate commitment of Header
• Submit Header and Data with submit -> height
• Get Proof for Header and Data with height and commitment
• Do light clients verify inclusion by commitment?

I thought inclusion was supposed to be verified by proof for now as we gossip it over rollkit p2p.

[Wondertan]

Submit receives a variadic Blob argument, which is effectively a slice.

[Wondertan]

See example: https://gobyexample.com/variadic-functions:

[Wondertan]

I thought inclusion was supposed to be verified by proof for now as we gossip it over rollkit p2p.

You cannot verify inclusion just with Blob. You need additional context like DAHeader. The Included provides you this

[Wondertan]

Ah, I see the confusion. The Included does not have Proof as param. Fixed

[nashqueue]

LGTM

[distractedm1nd]

Idea: would be helpful to add a Commitment method on the interface that returns the commitment of the blob. This would be helpful for using the blob module over the RPC CLI, and would be a good intermediate step for client implementations that haven't ported the logic for calculating commitments yet.

[Wondertan]

The cli/cmd/client logic can compute the commitment locally. Not on the remote node.

[Wondertan]

The principle for the APIs we have is that they are statefull. Stateless operations that don't require any special state/functionality from the node can be and should be done outside of the API.

[musalbas]

What's the difference between that and this (from original post)?

// Commitment computes the commitment of the Blob. 
// It is computed by splitting the blob into shares and building the Merkle subtree to be included after Submit.
func (b *Blob) Commitment() Commitment

[Wondertan]

Ryan proposed Commitment to be a method of BlobModule in addition to the method of the actual Blob(what you’ve shared)

[rootulp]

[non-blocking] I think it's a bit cumbersome that the following methods accept as an argument namespace and commitment. Given a commitment is sufficient to uniquely identify a blob, it seems a bit easier for a caller to only specify the commitment and not both the namespace and commitment. Given this is a UX concern and the caller likely knows the namespace they are querying for, this issue isn't blocking and could be considered a future improvement.

	// Get retrieves the blob by commitment. 
-	Get(context.Context, uint64, namespace.ID, Commitment) (Blob, error)
+	Get(context.Context, uint64, Commitment) (Blob, error) 

        // GetProof gets the Proof for a blob if it exists at the given height under the namespace. 
-        GetProof(context.Context, height uint64, namespace.ID, Commitment) (Proof, error)
+        GetProof(context.Context, height uint64, Commitment) (Proof, error)

	// Included checks whether a blob's given commitment(Merkle subtree root) is included at the given height under a particular namespace.
	// Useful for light clients wishing to check the inclusion of the data.
-	Included(context.Context, height uint64, namespace.ID, Commitment, Proof) (bool, error)
+ 	Included(context.Context, height uint64, Commitment, Proof) (bool, error)

[Wondertan]

Can we make commitment to include namespace?

[vgonkivs]

No, afaik it's not possible(currently)

[rootulp]

I think the share commitment could include the namespace. The last step of creating a share commitment is the step that loses the namespace field (i.e. generating a binary Merkle tree where the leaves of the tree are NMT subtree roots). So it seems possible for that step to prepend the namespace ID from the subtree roots (remember all the subtree roots will share a min/max namespace) to the resulting Merkle root. Note: this has the side-effect that the commitment size changes from 32 bytes to 65 bytes (1 byte for namespace version + 32 bytes for namespace ID + 32 bytes for original Merkle root).

[vgonkivs]

In order to verify the Proof, we will also need rowRoots(to verify nmt.Proofs against them), so I'd suggest extending BlobModule interface with the:
Verify(Blob, Proof)(bool)

[vgonkivs]

That's true, but then rollkit will need to get DAH first...

[Wondertan]

Why do we need this as separate method instead of sub operation of Included?

[vgonkivs]

To ensure that the Proof corresponds to the Blob.

[vgonkivs]

If it's not needed, I'm ok with sub-operation inside Included

[Wondertan]

I am not sure we need this a "sub operation" as well.

Included needs to verify inclusion of the commitment, not the blob(that's an important subtlety we found during on-site).

Included doesn't need to know the blob

[evan-forbes]

leaving a comment about paying for gas fees when submitting blobs here after discussing with @vgonkivs:

We likely want the user to have some input that determines the price of gas. If we have that, then we can estimate the gas limit, and then determine the fee. Both of those are then signed over. This simplifies the api, because the user only enters one thing, instead of fee and gas limit, which are both difficult to calculate.

Determining gas price could be as simple as the user specifying a direct price, or more complex where we estimate what the lowest gas price is to achieve some outcome, such as getting included in the next block, or getting included in any of the next 10 blocks etc.

[nashqueue]

How can you encompass the fee and gaslimit in one parameter?
It feels like estimation should be easy as you can simulate sparse shares construction and calculate the resulting cost.
Maybe also just a formula like gas(bytes) = gas_per_share * shares(bytes) and
shares(bytes)= roundup [ |bytes - (512 - first_share_overhead )| / (512 - share_overhead)]
As I understand, we only support one signature type right now? So Transactions and PFB Transactions should be fixed. The only variable is how many blobs are in one pfb transaction. I bet I am missing something here like multi Transactions.

Another fun idea: Make a TWAP for Gas.

[cmwaters]

There's a simulateTx endpoint on consensus nodes that could be used to calculate the gasLimit. As I mentioned elsewhere, the problem with using an offline formula is that the gas_per_blob_byte is a parameter that could change via governance. I think the ideal solution would be to have gas refunds (the excess gas goes back to your account).

As to gas prices, we could start by using the DefaultMinGasPrice set in celestia-app. In the future it would be nice to have it as an endpoint that is exposed (or perhaps returned in the SimulateTx call) since the value could fluctuate.