Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Application: Subcoin #2304

Merged
merged 3 commits into from
Jul 10, 2024
Merged

Application: Subcoin #2304

merged 3 commits into from
Jul 10, 2024

Conversation

liuchengxu
Copy link
Contributor

Project Abstract

Subcoin is a Bitcoin full node in Substrate. The primary goal of this grant is to implement a prototype of a Bitcoin node that is capable of syncing the Bitcoin network from the genesis to the chain tip using the advanced state sync strategy offered by the Substrate framework.

Grant level

  • Level 1: Up to $10,000, 2 approvals
  • Level 2: Up to $30,000, 3 approvals
  • Level 3: Unlimited, 5 approvals (for >$100k: Web3 Foundation Council approval)

Application Checklist

  • The application template has been copied and aptly renamed (project_name.md).
  • I have read the application guidelines.
  • Payment details have been provided (Polkadot AssetHub (DOT, USDC & USDT) address in the application and bank details via email, if applicable).
  • I understand that 30% of each milestone will be paid in vested DOT, to the Polkadot address listed in the application.
  • I am aware that, in order to receive a grant, I (and the entity I represent) have to successfully complete a KYC/KYB check.
  • The software delivered for this grant will be released under an open-source license specified in the application.
  • The initial PR contains only one commit (squash and force-push if needed).
  • The grant will only be announced once the first milestone has been accepted (see the announcement guidelines).
  • I prefer the discussion of this application to take place in a private Element/Matrix channel. My username is: @_______:matrix.org (change the homeserver if you use a different one)

@github-actions github-actions bot added the admin-review This application requires a review from an admin. label May 13, 2024
Copy link
Collaborator

@takahser takahser left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @liuchengxu

Thanks for applying again to our grants program (previous application see here, for those unaware).

I see that you've pivoted to developing a Bitcoin full node rather than a light client, when compared to your previous application. While I think this is a very interesting idea, I'm not fully convinced that it would greatly benefit the Polkadot Ecosystem. If you disagree, please explain why.

@takahser takahser added the changes requested The team needs to clarify a few things first. label May 21, 2024
@takahser takahser self-assigned this May 21, 2024
@liuchengxu
Copy link
Contributor Author

@takahser Thanks for your feedback. I'd like to address your concerns and clarify the broader vision for this project.

While this project might seem indirectly related to Polkadot at first glance, its implications are quite significant for the Substrate framework (polkadot-sdk) as a whole, and by extension, Polkadot.

By leveraging Substrate’s advanced state sync strategy, we can achieve faster and more efficient full node implementations. This not only benefits Bitcoin by increasing its decentralization but also demonstrates the power and flexibility of the Substrate framework, attracting more developers to the polkadot-sdk ecosystem, which in turn strengthens Polkadot’s position in the blockchain space.

@liuchengxu
Copy link
Contributor Author

I also want to add that by integrating a Bitcoin full node in Substrate, new possibilities are opened up for developers to build on Substrate with integrated Bitcoin functionalities, enhancing the overall utility and appeal of Bitcoin-related projects in the Polkadot ecosystem. For instance, the existing BTC bridge projects like ChainX and interlay using a BTC light client both require the role of relayer, with the Bitcoin networking feature developed in this application, we can directly pull blocks and transmit the transaction to the nodes of the Bitcoin network, without any need for intermediaries that might censor them.

@liuchengxu liuchengxu requested a review from takahser May 23, 2024 02:47
@PieWol
Copy link
Member

PieWol commented May 28, 2024

Hey @liuchengxu, thanks for the application. I would be interested in what you deem the biggest challenges when implementing this decentralized bitcoin full node.

@liuchengxu
Copy link
Contributor Author

@PieWol Thank you for your comment. This is an interesting point. The biggest challenge so far is achieving block execution performance comparable to other leading Bitcoin full node implementations. The state sync feature offered by Subcoin, while advantageous, introduces additional overhead due to the state trie, which significantly impacts block execution performance.

Based on my local experiments, after processing 300,000 blocks, the block execution becomes increasingly slower, executing a single transaction takes over one millisecond, and the average block time exceeds one second. Consequently, the initial full block sync required by bootstrap nodes may take several days, which is considerably longer than the best performance of around 8+ hours, as detailed in this post.

There are ongoing efforts to enhance Merkle trie performance, such as the work being done by thrumdev/nomt. These improvements could significantly help on this front. I will also focus on performance optimization once the prototype is fully developed, although not every Subcoin node has to do the full block sync.

@takahser
Copy link
Collaborator

By leveraging Substrate’s advanced state sync strategy, we can achieve faster and more efficient full node implementations. This not only benefits Bitcoin by increasing its decentralization but also demonstrates the power and flexibility of the Substrate framework, attracting more developers to the polkadot-sdk ecosystem, which in turn strengthens Polkadot’s position in the blockchain space.

While I agree that there might be a benefit for Substrate and the Polkadot SDK, I think the benefits are rather indirect than direct. Personally, I don't think it's a strong enough case to get support through the grants program.

For instance, the existing BTC bridge projects like ChainX and interlay using a BTC light client both require the role of relayer, with the Bitcoin networking feature developed in this application, we can directly pull blocks and transmit the transaction to the nodes of the Bitcoin network, without any need for intermediaries that might censor them.

That sounds interesting indeed. Does that mean the interlay relayers are centralised? Could you explain both mechanisms in detail, so we can get a better understanding?

@liuchengxu
Copy link
Contributor Author

That sounds interesting indeed. Does that mean the interlay relayers are centralised? Could you explain both mechanisms in detail, so we can get a better understanding?

The relayers of a Bitcoin bridge have two main tasks: a) They listen to the Bitcoin network, submit Bitcoin headers, and deposit transactions to the target chain (such as Interlay or ChainX) as needed. b) They broadcast withdrawal transactions from the target chain to the Bitcoin network.

The relayers for Interlay and ChainX operate as peripheral programs. They connect to a remote Bitcoin Core node using RPC. This setup theoretically allows for a permissionless system since the code is open-source, meaning anyone can build the program and run a relayer. However, in practice, the situation tends to be more centralized. Despite the open-source nature, the motivation to run the relayer program is often low, leading to a scenario where only the project builders are actively maintaining the relayers. This creates a single point of failure, as the relayer role is effectively centralized. If all the relayers go offline, the chain is paused.

In contrast, ICP offers a superior integration by implementing the Bitcoin network at the protocol level. ICP nodes can directly communicate with the Bitcoin P2P network without the need for an additional peripheral program, every ICP node serves as a relayer. The Bitcoin network crate developed in Subcoin will facilitate the Bitcoin network integration into a Substrate node.

@takahser
Copy link
Collaborator

In contrast, ICP offers a superior integration by implementing the Bitcoin network at the protocol level. ICP nodes can directly communicate with the Bitcoin P2P network without the need for an additional peripheral program, every ICP node serves as a relayer. The Bitcoin network crate developed in Subcoin will facilitate the Bitcoin network integration into a Substrate node.

Ohh, so the pallet basically directly encapsulates the bitcoin light client, similar to how BEEFY can bridge with EVM chains, right? But unidirectional, meaning we can bridge BTC to your substrate-based chain and back, but we can't bridge DOT or your native token to Bitcoin. Is that correct?

@liuchengxu
Copy link
Contributor Author

Ohh, so the pallet basically directly encapsulates the bitcoin light client, similar to how BEEFY can bridge with EVM chains, right? But unidirectional, meaning we can bridge BTC to your substrate-based chain and back, but we can't bridge DOT or your native token to Bitcoin. Is that correct?

That's correct. The on-chain pallet of interlay/chainx consists of a Bitcoin light client and the logic of processing BTC transfers. The bridge focuses on BTC only, there is no reason to bridge DOT or any other native token to Bitcoin anyway, it's not suitable for busy on-chain activities after all :P

@takahser
Copy link
Collaborator

The bridge focuses on BTC only, there is no reason to bridge DOT or any other native token to Bitcoin anyway, it's not suitable for busy on-chain activities after all :P

Makes sense. Yes, I tend to agree with that lol.

What are your long-term plans for this project, in case we decide to fund it?

@liuchengxu
Copy link
Contributor Author

@takahser Regarding the long-term plan for this project, I have several goals. First and foremost, I aim to make its performance, particularly in syncing, comparable to other leading Bitcoin full node implementations. This will ensure it becomes a viable option for running a Bitcoin node.

Secondly, I want to identify and address use cases for Subcoin. I have noticed some complaints that Bitcoin developers are currently facing with Bitcoin Core, and this project could potentially offer solutions to some of these inconveniences.

Lastly, I plan to seek financial support for its development and maintenance in the long run to ensure its sustainability. Your agreement and support would greatly benefit the success of this project as an endorsement.

@PieWol
Copy link
Member

PieWol commented May 31, 2024

@PieWol Thank you for your comment. This is an interesting point. The biggest challenge so far is achieving block execution performance comparable to other leading Bitcoin full node implementations. The state sync feature offered by Subcoin, while advantageous, introduces additional overhead due to the state trie, which significantly impacts block execution performance.

Based on my local experiments, after processing 300,000 blocks, the block execution becomes increasingly slower, executing a single transaction takes over one millisecond, and the average block time exceeds one second.

Hey @liuchengxu ,
thank you for the insights and the article you linked. So basically unless there is a solution found to the state trie problem the whole project is infeasable as we have steadily growing times to sync a block? How can this issue be fixed so that the chain is able to scale with the increasing bitcoin block height?

@liuchengxu
Copy link
Contributor Author

@PieWol Not necessarily. As I mentioned earlier, this issue primarily affects the Subcoin bootstrap node. However, not everyone needs to run a Subcoin bootstrap node using the full sync method. Newly joined nodes can leverage state sync to sync to the tip of the network quickly. Therefore, most users can still benefit from Subcoin's fast sync feature even if the state trie problem persists. The Bitcoin state sync in a decentralized manner is the Subcoin's most significant contribution.

Based on current experimental estimations, Subcoin's initial full sync performance is unlikely to be the worst, even with the additional state trie overhead. Other implementations like BTCD and Libbitcoin Server also take several days to complete the initial sync.

@liuchengxu
Copy link
Contributor Author

Hey, @takahser. No rush, but is there anything I can do to help move this PR forward?

@takahser
Copy link
Collaborator

@liuchengxu sorry for the delay, have been busy recently and I'm now catching up with unread notifications.

Secondly, I want to identify and address use cases for Subcoin. I have noticed some complaints that Bitcoin developers are currently facing with Bitcoin Core, and this project could potentially offer solutions to some of these inconveniences.

Do you happen to have any links to share here?

@liuchengxu
Copy link
Contributor Author

Do you happen to have any links to share here?

I don't have any specific links at the moment. My observations are based on discussions with developers within my network, which are wallet-related topics if I recall correctly. I anticipate that we'll see some specific demand from developers once we achieve a solid implementation in Substrate.

@semuelle semuelle added ready for review The project is ready to be reviewed by the committee members. and removed changes requested The team needs to clarify a few things first. labels Jun 12, 2024
@liuchengxu
Copy link
Contributor Author

@PieWol I did some Bitcoin-specific optimization within the substrate framework recently, mostly from the conventional engineering aspects like using an in-memory backend and off-runtime execution, the initial full sync performance of the bootstrap node has been significantly improved by more than 5x, and now the initial full sync is expected to take less than one day. This is not the end, chances are great that there will be more optimizations in the future, even if the state trie issue persists.

Ping @takahser I'm happy to chat offline if needed.

@PieWol
Copy link
Member

PieWol commented Jun 26, 2024

Hello @liuchengxu ,
I would like to have a chat with you to better understand the application. Could you please send a meeting invite for a short meeting to piet@web3.foundation if you are interested? Thanks.

@liuchengxu
Copy link
Contributor Author

Hey @PieWol You should have received the invite in the email, feel free to suggest another time if that does not work for you, I'm pretty flexible.

@takahser
Copy link
Collaborator

@liuchengxu sorry for the delay here, I was planning to review this in more depth, but I think a call sounds good as well. I'm planning to join the call as well. 👍

- Removed sending test transactions as it does not make sense in
  Subcoin.
- Added Bitcoin Transaction Broadcasting deliverable in M3.
- Updated some crate names based on the latest development.
@liuchengxu
Copy link
Contributor Author

@PieWol @takahser Thanks a lot for the meeting! The application has been updated with minor changes based on the offline discussion, please see the commit message for the main points. Let me know if anything is still missing.

Copy link
Collaborator

@takahser takahser left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@liuchengxu thanks for your time in today's call. Just one last question: Could you add more details on how a parachain is going to be able to interact with the bitcoin network using your client?

@liuchengxu
Copy link
Contributor Author

@PieWol @takahser Please help move this PR forward when you have time. I'm more than happy to have another call if you have any further questions. Thank you!

takahser
takahser previously approved these changes Jul 2, 2024
Copy link
Collaborator

@takahser takahser left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@liuchengxu thanks for your time today. Given the bitcoin blocks are wrapped and passed to the runtime's execute_block function and any Substrate-based chain can re-use it, I'm happy to add my approval as well.

PieWol
PieWol previously approved these changes Jul 2, 2024
Copy link
Member

@PieWol PieWol left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for explaining your project further in our meetings. I'm approving it.

Copy link
Member

@semuelle semuelle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @liuchengxu, welcome back. Thanks for the application. I am also happy to support the project. I only have one request: Beside the unit tests, could you also provide tutorials for each milestone, e.g. how to author and send transactions?

Are you familiar with Tuxedo? You might be able to leverage some of their work.

@liuchengxu
Copy link
Contributor Author

Beside the unit tests, could you also provide tutorials for each milestone, e.g. how to author and send transactions?

Hey @semuelle This was deliberately removed in 8823785 as authoring and sending transactions does not make a lot of sense here. In the meetings with PieWol and takahser, I clarified that Subcoin does not expose any callable extrinsics to users, we'll only support broadcasting a constructed Bitcoin transaction to Bitcoin network via subcoin RPC in this application, as one deliverable in M3. The core of this application is to implement a Bitcoin full node that can do an initial full sync from the Bitcoin network and provide the fast sync feature for the newly joined nodes. Constructing a Bitcoin transaction is a feature of a Bitcoin wallet, which is out of the scope for this application.

Tuxedo looks interesting, but with a quick look, we have much different goals. Subcoin is essentially a replica of the Bitcoin chain, the subcoin runtime is a minimalist Substrate runtime, containing only one storage item for the UTXO set, while Tuxedo primarily focuses on developing a brand new chain using the UTXO model. Thanks for the link anyway. We'll see if there is anything we could leverage and collaborate.

Copy link
Member

@semuelle semuelle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we'll only support broadcasting a constructed Bitcoin transaction to Bitcoin network via subcoin RPC in this application

Thanks for the quick reply. That's fine. I'm just looking for ways to test the deliverables beyond unit tests. But if that's only available in M3, then the spec of 2.0b doesn't make sense ("spin up one of our Substrate nodes and send test transactions"). It looks like all 0b specs are still the template text, could you review them?

@liuchengxu liuchengxu dismissed stale reviews from PieWol and takahser via 3991485 July 3, 2024 15:42
@liuchengxu
Copy link
Contributor Author

Good catch! @semuelle That's the leftover from the last commit. I have fixed it in the lastest push. We'll provide the instructions to run a Subcoin node, but no sending test transactions.

Copy link
Member

@semuelle semuelle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update. Happy to approve.

Copy link
Collaborator

@takahser takahser left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re-approving

@liuchengxu liuchengxu requested a review from PieWol July 5, 2024 07:22
Copy link
Member

@PieWol PieWol left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for refining the deliverables @semuelle and @liuchengxu.

@takahser takahser merged commit acbf5c8 into w3f:master Jul 10, 2024
7 of 8 checks passed
Copy link
Contributor

Congratulations and welcome to the Web3 Foundation Grants Program! Please refer to our Milestone Delivery repository for instructions on how to submit milestones and invoices, our FAQ for frequently asked questions and the support section of our README for more ways to find answers to your questions.

Before you start, take a moment to read through our announcement guidelines for all communications related to the grant or make them known to the right person in your organisation. In particular, please don't announce the grant publicly before at least the first milestone of your project has been approved. At that point or shortly before, you can get in touch with us at grantsPR@web3.foundation and we'll be happy to collaborate on an announcement about the work you’re doing.

Also, if you haven't yet, consider signing up for the Polkadot Alpha Program. The program offers plenty of resources for people building on Polkadot. Lastly, please remember to let us know in case you run into any delays or deviate from the deliverables in your application. You can either leave a comment here or directly request to amend your application via PR. We wish you luck with your project! 🚀

@semuelle
Copy link
Member

Hi @liuchengxu, could you fill out the KYC form (assuming you are applying as an individual, otherwise please use this form)?

@liuchengxu
Copy link
Contributor Author

Hey @semuelle I have finished the form and the verification was successful.

@semuelle
Copy link
Member

Confirmed!

@semuelle semuelle removed the admin-review This application requires a review from an admin. label Jul 11, 2024
@PieWol PieWol mentioned this pull request Aug 26, 2024
12 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ready for review The project is ready to be reviewed by the committee members.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants