Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EVM-660 Use batch write inside blockchain.go (LevelDB/memory) #1569

Merged
merged 20 commits into from
Jul 5, 2023

Conversation

jelacamarko
Copy link
Contributor

@jelacamarko jelacamarko commented May 30, 2023

Description

  • In the previous implementation, WriteFullBlock and other methods of Blockchain structure wrote each part of the block separately. However, with the changes introduced in this PR, everything is written together as a single batch. This can improve performance and efficiency when writing data
  • The KeyValueStorage is focused on read operations only + it provides a method [func (s *KeyValueStorage) NewBatch() Batch ] to create a concrete implementation of a Batch (memory or LevelDB)
  • The actual writing is handled by the BatchWriter, which holds the concrete Batch interface implementation. It contains methods such as PutHeadHash or PutHeadNumber etc that are putting desired changes into a batch queue. However, these methods do not immediately change the storage. The changes are only saved to the actual database or memory when the WriteBatch method is called.

LevelDB options that are changed:

OpenFilesCacheCapacity = DefaultHandles
BlockCacheCapacity = DefaultCache / 2 * opt.MiB
WriteBuffer = DefaultCache / 4 * opt.MiB

LevelDB options are set to the same values that go-ethereum uses.

DefaultHandles = 256
DefaultCache = 256

Changes include

  • Bugfix (non-breaking change that solves an issue)
  • Hotfix (change that solves an urgent issue, and requires immediate attention)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (change that is not backwards-compatible and/or changes current functionality)

Breaking changes

Please complete this section if any breaking changes have been made, otherwise delete it

Checklist

  • I have assigned this PR to myself
  • I have added at least 1 reviewer
  • I have added the relevant labels
  • I have updated the official documentation
  • I have added sufficient documentation in code

Testing

  • I have tested this code with the official test suite
  • I have tested this code manually

Manual tests

Please complete this section if you ran manual tests for this functionality, otherwise delete it

Documentation update

Please link the documentation update PR in this section if it's present, otherwise delete it

Additional comments

Please post additional comments in this section if you have them, otherwise delete it

@github-actions
Copy link

github-actions bot commented Jun 1, 2023

CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅

@jelacamarko jelacamarko self-assigned this Jun 1, 2023
blockchain/storage/leveldb/batch.go Outdated Show resolved Hide resolved
blockchain/storage/leveldb/leveldb.go Show resolved Hide resolved
blockchain/storage/leveldb/leveldb.go Outdated Show resolved Hide resolved
blockchain/storage/memory/batch.go Show resolved Hide resolved
blockchain/storage/batch.go Outdated Show resolved Hide resolved
@Stefan-Ethernal
Copy link
Collaborator

It would be probably interesting to implement some benchmark tests and see how much gain we get by using batch updates.

Copy link
Collaborator

@goran-ethernal goran-ethernal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Except from the small comments, looks good.

@jelacamarko jelacamarko requested a review from a team June 22, 2023 08:52
@igorcrevar igorcrevar self-requested a review July 3, 2023 16:25
@igorcrevar igorcrevar added the feature New update to Polygon Edge label Jul 4, 2023
@igorcrevar igorcrevar self-assigned this Jul 4, 2023
@igorcrevar igorcrevar changed the title Use LevelDB batch write for WriteFullBlock EVM-660 Use batch write inside blockchain.go (LevelDB/memory) Jul 4, 2023
@igorcrevar igorcrevar requested review from igorcrevar and removed request for igorcrevar July 4, 2023 09:02
Copy link
Contributor

@vcastellm vcastellm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM this should bring great improvement.

@igorcrevar igorcrevar merged commit 2d35db6 into develop Jul 5, 2023
7 checks passed
@github-actions github-actions bot locked and limited conversation to collaborators Jul 5, 2023
@Stefan-Ethernal Stefan-Ethernal deleted the Batch_POC branch July 8, 2023 06:14
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
feature New update to Polygon Edge
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants