Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: chain import: don't walk to genesis - 2-3x faster snapshot import #11446

Merged
merged 3 commits into from
Nov 25, 2023

Conversation

magik6k
Copy link
Contributor

@magik6k magik6k commented Nov 22, 2023

Related Issues

This cherrypicks 921a3a0 from #10821

Proposed Changes

  • Snapshot cars are written from chain top to bottom
  • We know what the top block header is because that's the car root
  • We know that the links in that block header will lead us to the correct parent
  • So when we see parents of the last seen block, we record those
  • Snapshots contain all block headers to genesis, so if we repeat this process for the whole import we never have to read a single block to find the genesis block
  • This way we can instantly set the correct genesis after snapshot import without having to walk there, saving roughly 17 000 000 useless reads from chainstore
    • (This is what causes the hand after chain import after the 'flushing validation cache' log)
    • For me it makes the mainnet snapshot import 2-3x faster

Additional Info

Checklist

Before you mark the PR ready for review, please make sure that:

  • Commits have a clear commit message.
  • PR title is in the form of of <PR type>: <area>: <change being made>
    • example: fix: mempool: Introduce a cache for valid signatures
    • PR type: fix, feat, build, chore, ci, docs, perf, refactor, revert, style, test
    • area, e.g. api, chain, state, market, mempool, multisig, networking, paych, proving, sealing, wallet, deps
  • If the PR affects users (e.g., new feature, bug fix, system requirements change), update the CHANGELOG.md and add details to the UNRELEASED section.
  • New features have usage guidelines and / or documentation updates in
  • Tests exist for new functionality or change in behavior
  • CI is green

@magik6k magik6k requested a review from a team as a code owner November 22, 2023 13:14
chain/store/snapshot.go Outdated Show resolved Hide resolved
chain/store/snapshot.go Show resolved Hide resolved
chain/store/snapshot.go Show resolved Hide resolved
}
}

if tailBlock.Height != 0 {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about the json genesis thing? (I assume you've tested this, I just wanted to make sure we're not missing something)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The first block with the BlockHeader type is at height 0, so here we just check for that - https://filfox.info/en/block/bafy2bzacecnamqgqmifpluoeldx7zzglxcljo6oja4vrmtj7432rphldpdmm2

@Stebalien Stebalien requested a review from hsanjuan November 22, 2023 14:49
@Stebalien Stebalien merged commit 231b207 into master Nov 25, 2023
87 checks passed
@Stebalien Stebalien deleted the feat/no-import-genesis-walk branch November 25, 2023 21:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants