Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(conductor, relayer)!: batch multiple Sequencer blocks to save on Celestia fees #1045

Merged
merged 12 commits into from
May 8, 2024

Conversation

SuperFluffy
Copy link
Member

@SuperFluffy SuperFluffy commented May 6, 2024

Summary

Batch mutiple Sequencer blocks into single Celestia blobs to reduce fee payments.

Background

Until now, each Sequencer block was turned into multiple blobs (one for overall sequencer block metadata, and one blob per rollup that had transactiosn in the sequencer block). This wasn't as efficient as it could be because the new compression scheme introduced in #1006 can only come to bear with more bytes to compress.

Relayer will collect sequencer blocks up to a total (compressed) size of 1MB (1/2 of the current max of 2MB that Celestia blocks can be).

Changes

  • Introduce protobuf messages astria.sequencerblock.v1alpha1.CelestiaHeaderList and astria.sequencerblock.v1alpha1.CelestiaRollupDataList
  • Rename astria.sequencerblock.v1alpha1.CelestiaRollupBlob to astria.sequencerblock.v1alpha1.CelestiaRollupData
  • Rename astria.sequencerblock.v1alpha1.CelestiaSequencerBlob to astria.sequencerblock.v1alpha1.CelestiaHeader
  • Collect Sequencer Blocks into the *List protobuf messages before posting them to Celestia (instead of splitting up each Sequencer block into mutiple blobs and posting them one by one).

Testing

Add unit tests around the next submission aggregation logic. Update conductor blackbox tests.

Metrics

  • CELESTIA_PAYLOAD_CREATION_LATENCY: histogram with microsecond units to track the time it takes to create a payload of Celestia blobs (encoding + compressing all protobufs)
  • metrics for reporting compression ratio and total compressed payload size were moved from the payload/blob construction phase to the submission phase.

Breaking Changelist

  • Relayer and Conductor write/read new protobuf messages to/from Celestia.

Related Issues

Closes #1042
Closes #1049

@github-actions github-actions bot added conductor pertaining to the astria-conductor crate proto pertaining to the Astria Protobuf spec sequencer-relayer pertaining to the astria-sequencer-relayer crate labels May 6, 2024
@SuperFluffy SuperFluffy changed the title Superfluffy/batched celestia feat(conductor, relayer)!: post batches of Sequencer blocks to Celestia May 6, 2024
@SuperFluffy SuperFluffy changed the title feat(conductor, relayer)!: post batches of Sequencer blocks to Celestia feat(conductor, relayer)!: batch multiple Sequencer blocks to save on Celestia fees May 6, 2024
@SuperFluffy SuperFluffy force-pushed the superfluffy/batched-celestia branch from 7bd6e3a to 1f753bc Compare May 6, 2024 22:05
@SuperFluffy SuperFluffy marked this pull request as ready for review May 6, 2024 22:06
@SuperFluffy SuperFluffy requested review from a team as code owners May 6, 2024 22:06
@SuperFluffy SuperFluffy requested review from noot and Fraser999 May 6, 2024 22:06
@SuperFluffy SuperFluffy removed the request for review from noot May 6, 2024 22:06
Copy link
Contributor

@Fraser999 Fraser999 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - mostly just nitpicks and a couple of questions.

crates/astria-conductor/src/celestia/convert.rs Outdated Show resolved Hide resolved
crates/astria-conductor/src/celestia/convert.rs Outdated Show resolved Hide resolved
crates/astria-conductor/src/celestia/convert.rs Outdated Show resolved Hide resolved
crates/astria-sequencer-relayer/src/relayer/write/mod.rs Outdated Show resolved Hide resolved
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't ever loved the naming of these, and I think the new names make it if anything worse.

The core issue is that these are Celestia prefixed but the data has nothing to do with celestia. Especially the new CelestiaHeader is not great because Celestia has headers and have nothing to do with these.

I propose we get rid of the Celestia prefix entirely and have RollupData and SequencerBlockMetadata, and their list forms. The Sequencer Block one is hard because it's the SequencerBlockHeader plus some information. I think metadata encompasses this alright.

Copy link
Member Author

@SuperFluffy SuperFluffy May 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately RollupData was already taken in the same protobuf package. I chosen the following nomenclature instead in 401233a:

  • SubmittedMetadata: since the full type name is actually astria.sequencerblock.v1alpha1.SubmittedMetadata it's clear that this pertains to a sequencerblock.
  • SubmittedRollupData

@SuperFluffy
Copy link
Member Author

@joroshiba I have added a histogram metric CELESTIA_PAYLOAD_CREATION_LATENCY and changed the protobufs to SubmittedMetadata, SubmittedMetadataList, SubmittedRollupData, SubmittedRollupDataList.

@SuperFluffy SuperFluffy requested a review from joroshiba May 8, 2024 15:26
Copy link
Member

@joroshiba joroshiba left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

API approval, new names look good.

@SuperFluffy SuperFluffy added this pull request to the merge queue May 8, 2024
Merged via the queue into main with commit 00f6b13 May 8, 2024
39 checks passed
@SuperFluffy SuperFluffy deleted the superfluffy/batched-celestia branch May 8, 2024 15:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
conductor pertaining to the astria-conductor crate proto pertaining to the Astria Protobuf spec sequencer-relayer pertaining to the astria-sequencer-relayer crate
Projects
None yet
3 participants