Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dump IO Usage #473

Merged
merged 2 commits into from
Aug 19, 2022
Merged

Dump IO Usage #473

merged 2 commits into from
Aug 19, 2022

Conversation

pratikspatil024
Copy link
Member

@pratikspatil024 pratikspatil024 commented Jul 27, 2022

@pratikspatil024 pratikspatil024 force-pushed the block-stm-dumpio branch 2 times, most recently from 5b387d6 to 8ee6f6c Compare July 27, 2022 07:10
@codecov-commenter
Copy link

codecov-commenter commented Jul 27, 2022

Codecov Report

Merging #473 (8ee6f6c) into block-stm (ecedc65) will increase coverage by 0.04%.
The diff coverage is 65.11%.

@@              Coverage Diff              @@
##           block-stm     #473      +/-   ##
=============================================
+ Coverage      56.11%   56.15%   +0.04%     
=============================================
  Files            604      607       +3     
  Lines          70109    70171      +62     
=============================================
+ Hits           39341    39406      +65     
+ Misses         27368    27364       -4     
- Partials        3400     3401       +1     
Impacted Files Coverage Δ
core/parallel_state_processor.go 0.00% <0.00%> (ø)
core/state/statedb.go 62.41% <0.00%> (-0.89%) ⬇️
eth/tracers/api.go 26.89% <84.37%> (+4.09%) ⬆️
core/state_transition.go 88.16% <100.00%> (ø)
rpc/server.go 72.50% <0.00%> (-5.00%) ⬇️
p2p/enode/nodedb.go 70.46% <0.00%> (-3.11%) ⬇️
les/vflux/client/serverpool.go 76.76% <0.00%> (-1.66%) ⬇️
p2p/discover/v4_udp.go 75.65% <0.00%> (-1.45%) ⬇️
eth/protocols/snap/sync.go 70.45% <0.00%> (-1.28%) ⬇️
les/odr.go 89.02% <0.00%> (-1.22%) ⬇️
... and 29 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ecedc65...8ee6f6c. Read the comment docs.

core/state/statedb.go Outdated Show resolved Hide resolved
@pratikspatil024 pratikspatil024 force-pushed the block-stm-dumpio branch 2 times, most recently from 73f7e5d to c9390f8 Compare August 4, 2022 06:02
core/state/statedb.go Outdated Show resolved Hide resolved
core/state/statedb.go Outdated Show resolved Hide resolved
eth/tracers/api.go Outdated Show resolved Hide resolved
eth/tracers/api.go Outdated Show resolved Hide resolved
eth/tracers/api.go Outdated Show resolved Hide resolved
@cffls
Copy link
Contributor

cffls commented Aug 18, 2022

Looks like linter failed. Could you fix before merging?

@pratikspatil024
Copy link
Member Author

@cffls @JekaMas I am not getting the lint errors on my local machine. Also, these lines, where the CI is throwing an error is not present in the code.

Error: core/parallel_state_processor.go:137:1: cognitive complexity 32 of func `(*ParallelStateProcessor).Process` is high (> 30) (gocognit)
func (p *ParallelStateProcessor) Process(block *types.Block, statedb *state.StateDB, cfg vm.Config) (types.Receipts, []*types.Log, uint64, error) {
^
Error: core/parallel_state_processor.go:101:3: if statements should only be cuddled with assignments (wsl)
		if _, ok := task.statedb.MVReadMap()[string(task.result.BurntContractAddress.Bytes())]; ok {
		^
Error: core/parallel_state_processor.go:195:4: branch statements should not be cuddled if block has more than two lines (wsl)
			break

@pratikspatil024
Copy link
Member Author

pratikspatil024 commented Aug 18, 2022

@cffls @JekaMas I am not getting the lint errors on my local machine. Also, these lines, where the CI is throwing an error is not present in the code.

Error: core/parallel_state_processor.go:137:1: cognitive complexity 32 of func `(*ParallelStateProcessor).Process` is high (> 30) (gocognit)
func (p *ParallelStateProcessor) Process(block *types.Block, statedb *state.StateDB, cfg vm.Config) (types.Receipts, []*types.Log, uint64, error) {
^
Error: core/parallel_state_processor.go:101:3: if statements should only be cuddled with assignments (wsl)
		if _, ok := task.statedb.MVReadMap()[string(task.result.BurntContractAddress.Bytes())]; ok {
		^
Error: core/parallel_state_processor.go:195:4: branch statements should not be cuddled if block has more than two lines (wsl)
			break

Actually these changes were added in this commit in the base repo (block-atm). And I think because of that the linters are failing. These will be resolved, but one lint error which might need some thinking is. this one:

Error: core/parallel_state_processor.go:142:1: cognitive complexity 32 of func `(*ParallelStateProcessor).Process` is high (> 30) (gocognit)

@pratikspatil024 pratikspatil024 force-pushed the block-stm branch 2 times, most recently from d031cd2 to 680a63b Compare August 18, 2022 11:11
@cffls
Copy link
Contributor

cffls commented Aug 18, 2022

I guess it is okay to merge then, since some failed linters are caused by another commit. We can fix linter issues altogether in this PR: #490

@pratikspatil024 pratikspatil024 merged commit 0df5133 into block-stm Aug 19, 2022
cffls pushed a commit to cffls/bor that referenced this pull request Sep 20, 2022
cffls pushed a commit to cffls/bor that referenced this pull request Sep 28, 2022
pratikspatil024 added a commit that referenced this pull request Oct 7, 2022
* Create MVHashMap and use it StateDB

* Parallel state processor

* Move fee burning and tipping out of state transition to reduce read/write dependencies between transactions

* Re-execute parallel tasks when there is a read in coinbase or burn address

* Block-stm optimization

Added tests for executor and two major improvements:

1. Add a dependency map during execution. This will prevent aborted tasks from being sent for execution immedaitely after failure.
2. Change the key of MVHashMap from string to a byte array. This will reduce time to convert byte slices to strings.

* Remove cache from executor test

* added mvhashmap unit tests (with  as key)

* Shard mvhashmap to reduce the time spent in global mutex

* Skip applying intermediate states

* Dependency improvement

* added test for status

* Create MVHashMap and use it StateDB

* Parallel state processor

* Move fee burning and tipping out of state transition to reduce read/write dependencies between transactions

* Re-execute parallel tasks when there is a read in coinbase or burn address

* Txn prioritizer implemented using mutex map (#487)

* basic txn prioritizer implemented using mutex map

* Re-execute parallel tasks when there is a read in coinbase or burn address

* Re-execute parallel tasks when there is a read in coinbase or burn address

* using *sync.RWMutex{} in mutexMap

Co-authored-by: Jerry <jerrycgh@gmail.com>

* added getReadMap and getWriteMap (#473)

* Block-stm optimization

Added tests for executor and some improvements:

1. Add a dependency map during execution. This will prevent aborted tasks from being sent for execution immedaitely after failure.
2. Change the key of MVHashMap from string to a byte array. This will reduce time to convert byte slices to strings.
3. Use sync.Map to reduce the time spent in global mutex.
4. Skip applying intermediate states.
5. Estimate dependency when an execution fails without dependency information.
6. Divide execution task queue into two separate queues. One for relatively certain transactions, and the other for speculative future transactions.
7. Setting dependencies of Txs coming from the same sender before starting parallel execution.
8. Process results in their semantic order (transaction index) instead of the order when they arrive. Replace result channel with a priority queue.

* Do not write entire objects directly when applying write set in blockstm

* fixed a small bug in the Report function (#530)

* linters

Co-authored-by: Jerry <jerrycgh@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants