[R4R] performance improvement in many aspects #257

unclezoro · 2021-05-26T03:55:19Z

Description

This PR tries to improve the performance of BSC.

Rationale

Part of Block Processing can be parallel, and the layer of the snapshot is too deeper for the current implementation. This PR focus on:

Do BlockBody verification concurrently;
Do the calculation of intermediate root concurrently;
Commit the MPTs concurrently;
Preload accounts before processing blocks;
Make the snapshot layers configurable.
Reuse some objects to reduce GC.
Cherry-pick the RLP improvement from go-ethereum.
Other minor improvements.

Example

The testing result on an 8 core 2.5G Hz Linux VM:

The full sync speed on the main-net is 40% faster.
The execution of an BEP20 transfer is 50% faster.

Changes

New command flags: --triesInMemory

Preflight checks

build passed (make build)
tests passed (make test)
manual transaction test passed

Already reviewed by

...

Related issues

... reference related issue #'s here ...

yutianwu · 2021-07-14T10:40:50Z

eth/downloader/peer.go


+	go p.peer.RequestHeadersByNumber(from, count, 0, false)


why not use gopool here

It is used when clients doing fast sync and snap sync. Not related to performance

yutianwu · 2021-07-14T10:42:51Z

p2p/rlpx/rlpx.go

 	"github.com/ethereum/go-ethereum/crypto"
 	"github.com/ethereum/go-ethereum/crypto/ecies"
 	"github.com/ethereum/go-ethereum/rlp"
-	"github.com/golang/snappy"
-	"golang.org/x/crypto/sha3"
+	"github.com/oxtoacart/bpool"


yutianwu · 2021-07-15T03:44:08Z

core/blockchain.go

+		}
+		if len(accountsSlice) >= preLoadLimit {
+			objsChan := make(chan []*state.StateObject, runtime.NumCPU())
+			for i := 0; i < runtime.NumCPU(); i++ {


should we make sure that len(accountsSlice) > runtime.NumCPU()

Add one more condition: len(accountsSlice) > runtime.NumCPU()

core/blockchain.go

adithyaxx · 2021-07-28T06:26:05Z

Can we get this PR merged?

yvettep321

_thank you _

Description: This PR refers to the code optimization done by `BSC`, which mainly includes the following parts: 1. Do BlockBody verification concurrently. 2. Do the calculation of intermediate root concurrently. 3. Commit the MPTs concurrently. 4. Preload accounts before processing blocks. 5. Make the snapshot layers configurable. 6. Reuse some objects to reduce GC. 7. Add shared_pool for `stateDB` to improve cache usage. References: - bnb-chain/bsc#257 - bnb-chain/bsc#792 --------- Co-authored-by: j75689 <j75689@gmail.com> Co-authored-by: realowen <131384228+realowen@users.noreply.github.com>

Description This PR refers to the code optimization done by `BSC`, which mainly includes the following parts: 1. Do BlockBody verification concurrently. 2. Do the calculation of intermediate root concurrently. 3. Commit the MPTs concurrently. 4. Preload accounts before processing blocks. 5. Make the snapshot layers configurable. 6. Reuse some objects to reduce GC. 7. Add shared_pool for `stateDB` to improve cache usage. References - bnb-chain/bsc#257 - bnb-chain/bsc#792 --------- Co-authored-by: j75689 <j75689@gmail.com> Co-authored-by: realowen <131384228+realowen@users.noreply.github.com>

Description This PR refers to the code optimization done by `BSC`, which mainly includes the following parts: 1. Do BlockBody verification concurrently. 2. Do the calculation of intermediate root concurrently. 3. Commit the MPTs concurrently. 4. Preload accounts before processing blocks. 5. Make the snapshot layers configurable. 6. Reuse some objects to reduce GC. 7. Add shared_pool for `stateDB` to improve cache usage. References - bnb-chain/bsc#257 - bnb-chain/bsc#792 --------- Co-authored-by: j75689 <j75689@gmail.com>

accountTrieCache and storageTrieCache were introduced in this PR: bnb-chain#257, which is to improve performance. Actually the performance gain is quite limitted, as there is already dirty and clean cache for trie node. And after big merge, these 2 cache can not be used when PBSS is enabled. So remove these code to simplify the logic.

accountTrieCache and storageTrieCache were introduced in this PR: #257, which is to improve performance. Actually the performance gain is quite limited, as there is already dirty and clean cache for trie node. And after big merge, these 2 cache can not be used when PBSS is enabled. So remove these code to simplify the logic.

accountTrieCache and storageTrieCache were introduced in this PR: bnb-chain#257, which is to improve performance. Actually the performance gain is quite limited, as there is already dirty and clean cache for trie node. And after big merge, these 2 cache can not be used when PBSS is enabled. So remove these code to simplify the logic.

This is part of the optimizations from BSC, microbench show the change could introduce speed up of simple erc20 transfer scenario. The original PR of BSC: bnb-chain/bsc#257

This is part of the optimizations from BSC, microbench show the change could introduce ~20% speed up of simple erc20 transfer scenario. The original PR of BSC: bnb-chain/bsc#257

unclezoro force-pushed the improve_8 branch 13 times, most recently from 62904a0 to 5bdb7b4 Compare May 31, 2021 07:35

unclezoro force-pushed the improve_8 branch 17 times, most recently from 0a1cc10 to e750b5d Compare June 7, 2021 13:19

yutianwu reviewed Jul 14, 2021

View reviewed changes

yutianwu reviewed Jul 15, 2021

View reviewed changes

core/blockchain.go Outdated Show resolved Hide resolved

zfesd approved these changes Jul 19, 2021

View reviewed changes

fix can not repair

94e1505

unclezoro force-pushed the improve_8 branch from 38c7d26 to 94e1505 Compare July 20, 2021 12:25

address comments

8975587

unclezoro force-pushed the improve_8 branch from c4ffd75 to 8975587 Compare July 26, 2021 02:11

ifstatic approved these changes Jul 26, 2021

View reviewed changes

yutianwu approved these changes Jul 29, 2021

View reviewed changes

j75689 approved these changes Jul 29, 2021

View reviewed changes

unclezoro merged commit 2ce00ad into master Jul 29, 2021

unclezoro mentioned this pull request Jul 29, 2021

[R4R]prepare release 1.1.1-beta #334

Merged

3 tasks

unclezoro deleted the improve_8 branch July 29, 2021 11:18

j75689 mentioned this pull request Jul 30, 2021

BSC synchronization issues #338

Closed

j75689 linked an issue Jul 30, 2021 that may be closed by this pull request

BSC synchronization issues #338

Closed

yvettep321 reviewed Aug 31, 2021

View reviewed changes

j75689 added a commit to j75689/bsc that referenced this pull request Mar 24, 2022

delete unused cases, ref bnb-chain#257

3d5139f

brilliant-lx mentioned this pull request Oct 27, 2023

code: remove accountTrieCache and storageTrieCache #1949

Merged

sunny2022da mentioned this pull request Apr 10, 2024

feat: introduce hash cache to improve performance bnb-chain/op-geth#81

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[R4R] performance improvement in many aspects #257

[R4R] performance improvement in many aspects #257

unclezoro commented May 26, 2021 •

edited

Loading

yutianwu Jul 14, 2021

unclezoro Jul 22, 2021

yutianwu Jul 14, 2021

unclezoro Jul 22, 2021

yutianwu Jul 15, 2021

unclezoro Jul 22, 2021

adithyaxx commented Jul 28, 2021

yvettep321 left a comment

[R4R] performance improvement in many aspects #257

[R4R] performance improvement in many aspects #257

Conversation

unclezoro commented May 26, 2021 • edited Loading

Description

Rationale

Example

Changes

Preflight checks

Already reviewed by

Related issues

yutianwu Jul 14, 2021

Choose a reason for hiding this comment

unclezoro Jul 22, 2021

Choose a reason for hiding this comment

yutianwu Jul 14, 2021

Choose a reason for hiding this comment

unclezoro Jul 22, 2021

Choose a reason for hiding this comment

yutianwu Jul 15, 2021

Choose a reason for hiding this comment

unclezoro Jul 22, 2021

Choose a reason for hiding this comment

adithyaxx commented Jul 28, 2021

yvettep321 left a comment

Choose a reason for hiding this comment

unclezoro commented May 26, 2021 •

edited

Loading