Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(share): Shwap Bitwap composition #3421

Merged
merged 40 commits into from
Jun 26, 2024
Merged

Conversation

Wondertan
Copy link
Member

@Wondertan Wondertan commented May 21, 2024

Implements Bitswap composition based on CIP-19 by introducing three Block types for every Shwap container(Row, Sample, RowNamespaceData). Besides, this PR brings Block interfaces that generalize over those new blocks. This interface enables a slightly convoluted general-purpose Fetch function to get any Block type over Bitswap, abstracting all the complexities. Besides, the current architecture provides a horizontally scalable framework for the new blocks that makes it super easy to add new Shwap containers to our protocol, like for future Range and Blob containers.

This is like the 5th 6th 7th iteration of the composition design and the most straightforward one @walldiss and I could come up with the requirements we had.

Respective changes in the spec: celestiaorg/CIPs#159

@Wondertan Wondertan added the kind:feat Attached to feature PRs label May 21, 2024
@walldiss walldiss added the shwap label May 22, 2024
share/shwap/p2p/bitswap/bitswap.go Outdated Show resolved Hide resolved
share/shwap/namespace_data.go Outdated Show resolved Hide resolved
share/shwap/namespace_data.go Outdated Show resolved Hide resolved
share/shwap/p2p/bitswap/bitswap_test.go Outdated Show resolved Hide resolved
share/shwap/p2p/bitswap/bitswap_test.go Outdated Show resolved Hide resolved
share/shwap/p2p/bitswap/cid.go Outdated Show resolved Hide resolved
share/shwap/p2p/bitswap/cid.go Outdated Show resolved Hide resolved
@Wondertan Wondertan changed the base branch from shwap to shwap-types-improvements May 27, 2024 12:46
Base automatically changed from shwap-types-improvements to shwap May 27, 2024 16:21
@Wondertan Wondertan marked this pull request as ready for review May 30, 2024 21:13
share/shwap/p2p/bitswap/block.go Outdated Show resolved Hide resolved
share/shwap/p2p/bitswap/block_fetch.go Outdated Show resolved Hide resolved
share/shwap/p2p/bitswap/block_fetch.go Outdated Show resolved Hide resolved
share/shwap/p2p/bitswap/block_fetch_test.go Outdated Show resolved Hide resolved
share/shwap/p2p/bitswap/row_block.go Outdated Show resolved Hide resolved
share/shwap/p2p/bitswap/row_block_test.go Outdated Show resolved Hide resolved
share/shwap/p2p/bitswap/row_namespace_data_block.go Outdated Show resolved Hide resolved
share/shwap/p2p/bitswap/row_block.go Show resolved Hide resolved
share/shwap/p2p/bitswap/block_fetch_test.go Outdated Show resolved Hide resolved
share/shwap/p2p/bitswap/sample_block_test.go Outdated Show resolved Hide resolved
share/shwap/p2p/bitswap/block_fetch.go Outdated Show resolved Hide resolved
share/shwap/p2p/bitswap/block_fetch.go Outdated Show resolved Hide resolved
share/shwap/p2p/bitswap/block_fetch.go Outdated Show resolved Hide resolved
share/shwap/p2p/bitswap/block_fetch.go Show resolved Hide resolved
share/shwap/p2p/bitswap/block_fetch.go Outdated Show resolved Hide resolved
share/shwap/p2p/bitswap/block_store.go Outdated Show resolved Hide resolved
share/shwap/p2p/bitswap/block_store.go Outdated Show resolved Hide resolved
share/shwap/p2p/bitswap/block_store.go Outdated Show resolved Hide resolved
share/shwap/p2p/bitswap/block_store.go Show resolved Hide resolved
share/shwap/p2p/bitswap/row_block_test.go Outdated Show resolved Hide resolved
@Wondertan
Copy link
Member Author

Wondertan commented Jun 24, 2024

There are two more things:

  • Fetch should optionally be able to store Blocks in the Bookstore
    • We could do that outside but that would require going through Block conventions and Marshalling again
  • The reason we have to limit fetch to 1024 blocks is partially because of this issue
    • Reading through it, I realized that the current Fetch is inefficient. It fetches batches of blocks synchronously, one after another. Instead, we should be streaming new block requests as old ones are fulfilled
    • Will be added to Shwap hardening and optimizations tracking issue #3513 and left for later as not critical
  • Provide constructors for Bitswap Client and Server with parameters configured and adapted for our DA network
    • Will go in the next PR

Copy link
Member

@walldiss walldiss left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Elegancy avoiding complexity. Beautiful! 🚀

@Wondertan Wondertan merged commit c98daef into shwap Jun 26, 2024
27 checks passed
@Wondertan Wondertan deleted the shwap-bitswap-composition branch June 26, 2024 15:49
walldiss pushed a commit to walldiss/celestia-node that referenced this pull request Jul 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind:feat Attached to feature PRs shwap
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants