Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Revert "feat: chain: make chain tipset fetching 1000x faster" #10468

Closed
wants to merge 1 commit into from
Closed

Revert "feat: chain: make chain tipset fetching 1000x faster" #10468

wants to merge 1 commit into from

Conversation

magik6k
Copy link
Contributor

@magik6k magik6k commented Mar 14, 2023

Reverts #10423

Not sure why, probably due to massive lock contention on indexCacheLk this appears to freeze in initial sync for a long time.

Also there were reports that API calls get blocked while state compute is going on, which may be related to this PR.

Opening a revert PR so I can test that theory, ideally we can fix the original code.

@magik6k magik6k requested a review from a team as a code owner March 14, 2023 07:43
@magik6k
Copy link
Contributor Author

magik6k commented Mar 14, 2023

One of my blocked nodes:

goroutine 1315 [runnable]:
github.com/multiformats/go-multihash.encodeHash({0x64c3d20, 0xc080275c80}, 0xb220, 0xffffffffffffffff)
	/home/magik6k/.opt/go/pkg/mod/github.com/multiformats/go-multihash@v0.2.1/sum.go:50 +0x185
github.com/multiformats/go-multihash.Sum({0xc0218d2000, 0x432, 0x800}, 0x30?, 0xc00058f400?)
	/home/magik6k/.opt/go/pkg/mod/github.com/multiformats/go-multihash@v0.2.1/sum.go:29 +0x97
github.com/ipfs/go-cid.V1Builder.Sum({0xc0?, 0x1f01b3?, 0xc055e87f80?}, {0xc0218d2000?, 0x60?, 0xc042cbe4e0?})
	/home/magik6k/.opt/go/pkg/mod/github.com/ipfs/go-cid@v0.3.2/builder.go:60 +0x4a
github.com/filecoin-project/go-state-types/abi.cidBuilder.Sum({0x649fe40?}, {0xc0218d2000?, 0xc092387320?, 0xc0604f4000?})
	/home/magik6k/.opt/go/pkg/mod/github.com/filecoin-project/go-state-types@v0.10.0/abi/cid.go:42 +0x45
github.com/filecoin-project/lotus/chain/types.(*BlockHeader).ToStorageBlock(0xc091c07500?)
	/home/magik6k/github.com/filecoin-project/go-lotus/chain/types/blockheader.go:74 +0x54
github.com/filecoin-project/lotus/chain/types.(*BlockHeader).Cid(0x4891880?)
	/home/magik6k/github.com/filecoin-project/go-lotus/chain/types/blockheader.go:83 +0x19
github.com/filecoin-project/lotus/chain/types.NewTipSet({0xc0852a5b40?, 0x8, 0x8})
	/home/magik6k/github.com/filecoin-project/go-lotus/chain/types/tipset.go:132 +0x23f
github.com/filecoin-project/lotus/chain/store.(*ChainStore).LoadTipSet(0xc000752000, {0x64bd100?, 0xc044144fc0}, {{0xc05098f180?, 0x130?}})
	/home/magik6k/github.com/filecoin-project/go-lotus/chain/store/store.go:847 +0x145
github.com/filecoin-project/lotus/chain/store.(*ChainIndex).walkBack(0xc000562aa0, {0x64bd100, 0xc044144fc0}, 0x98?, 0x1d8b48)
	/home/magik6k/github.com/filecoin-project/go-lotus/chain/store/index.go:175 +0xed
github.com/filecoin-project/lotus/chain/store.(*ChainIndex).fillCache(0xc000562aa0, {0x64bd100, 0xc044144fc0}, {{0xc04f9f38c0?, 0xe4?}})
	/home/magik6k/github.com/filecoin-project/go-lotus/chain/store/index.go:132 +0x193
github.com/filecoin-project/lotus/chain/store.(*ChainIndex).GetTipsetByHeight(0xc000562aa0, {0x64bd100, 0xc044144fc0}, 0xc042cbe898?, 0x1)
	/home/magik6k/github.com/filecoin-project/go-lotus/chain/store/index.go:67 +0x17a
github.com/filecoin-project/lotus/chain/store.(*ChainStore).GetTipsetByHeight(0xc000752000, {0x64bd100, 0xc044144fc0}, 0x1, 0x7dcd86?, 0x0)
	/home/magik6k/github.com/filecoin-project/go-lotus/chain/store/store.go:1165 +0xac
github.com/filecoin-project/lotus/chain/stmgr.tryLookupTipsetState({0x64bd100, 0xc044144fc0}, 0xc000752000, 0xc0441ea840)
	/home/magik6k/github.com/filecoin-project/go-lotus/chain/stmgr/execute.go:77 +0x6c
github.com/filecoin-project/lotus/chain/stmgr.(*StateManager).TipSetState(0xc000790d80, {0x64bd100?, 0xc043f53b60?}, 0xc0441ea840)
	/home/magik6k/github.com/filecoin-project/go-lotus/chain/stmgr/execute.go:58 +0x6e5
github.com/filecoin-project/lotus/chain/stmgr.(*StateManager).setupGenesisVestingSchedule(0xc000790d80, {0x64bd100, 0xc043f53b60})
	/home/magik6k/github.com/filecoin-project/go-lotus/chain/stmgr/supply.go:43 +0x155
github.com/filecoin-project/lotus/chain/stmgr.(*StateManager).GetFilVested(0xc000790d80, {0x64bd100, 0xc043f53b60}, 0x28ee21)
	/home/magik6k/github.com/filecoin-project/go-lotus/chain/stmgr/supply.go:206 +0x15d
github.com/filecoin-project/lotus/chain/stmgr.(*StateManager).GetVMCirculatingSupplyDetailed(0x0?, {0x64bd100, 0xc043f53b60}, 0x28ee21, 0x26?)
	/home/magik6k/github.com/filecoin-project/go-lotus/chain/stmgr/supply.go:342 +0x65
github.com/filecoin-project/lotus/chain/stmgr.(*StateManager).GetVMCirculatingSupply(0xc042cbf0f0?, {0x64bd100?, 0xc043f53b60?}, 0x46969e0?, 0xc043f23b01?)
	/home/magik6k/github.com/filecoin-project/go-lotus/chain/stmgr/supply.go:333 +0x3b
github.com/filecoin-project/lotus/chain/vm.defaultFVMOpts({0x64bd100, 0xc043f53b60}, 0xc04384aa80)
	/home/magik6k/github.com/filecoin-project/go-lotus/chain/vm/fvm.go:250 +0x163
github.com/filecoin-project/lotus/chain/vm.NewFVM({0x64bd100?, 0xc043f53b60?}, 0xc04384aa80)
	/home/magik6k/github.com/filecoin-project/go-lotus/chain/vm/fvm.go:279 +0x2c
github.com/filecoin-project/lotus/chain/vm.NewVM({0x64bd100?, 0xc043f53b60?}, 0x9dbd001?)
	/home/magik6k/github.com/filecoin-project/go-lotus/chain/vm/vmi.go:41 +0x45
github.com/filecoin-project/lotus/chain/stmgr.(*StateManager).VMConstructor.func1({0x64bd100?, 0xc043f53b60?}, 0xc026208400?)
	/home/magik6k/github.com/filecoin-project/go-lotus/chain/stmgr/stmgr.go:444 +0x2e
github.com/filecoin-project/lotus/chain/consensus.(*TipSetExecutor).ApplyBlocks.func2({{0xc031508b70?, 0xc043f53b60?}}, 0x28ee21, 0x6410203e)
	/home/magik6k/github.com/filecoin-project/go-lotus/chain/consensus/compute_state.go:291 +0x302
github.com/filecoin-project/lotus/chain/consensus.(*TipSetExecutor).ApplyBlocks(0xc00072a1f8, {0x64bd100, 0xc000b7a7e0}, 0xc000790d80, 0x28ee20, {{0xc031508b70?, 0x28ee22?}}, {0xc000e98120, 0x4, 0x4}, ...)
	/home/magik6k/github.com/filecoin-project/go-lotus/chain/consensus/compute_state.go:365 +0xc7c
github.com/filecoin-project/lotus/chain/consensus.(*TipSetExecutor).ExecuteTipSet(0x64bd100?, {0x64bd100?, 0xc000b7a7b0?}, 0xc000790d80, 0xc03fc048c0, {0x0, 0x0}, 0x0?)
	/home/magik6k/github.com/filecoin-project/go-lotus/chain/consensus/compute_state.go:548 +0x7ef
github.com/filecoin-project/lotus/chain/stmgr.(*StateManager).TipSetState(0xc000790d80, {0x64bd090?, 0xc000056050?}, 0xc03fc048c0)
	/home/magik6k/github.com/filecoin-project/go-lotus/chain/stmgr/execute.go:62 +0x733
github.com/filecoin-project/lotus/chain/messagepool.(*mpoolProvider).GetActorAfter(0xc000728e80, {{0xc0426ae1c0?, 0xc00055e8e0?}}, 0xc000b9a2d0?)
	/home/magik6k/github.com/filecoin-project/go-lotus/chain/messagepool/provider.go:94 +0x21b
github.com/filecoin-project/lotus/chain/messagepool.(*MessagePool).getStateNonce(0xc0423f6d00, {0x64bd058?, 0xc041ceff40?}, {{0xc0426ae1c0?, 0x0?}}, 0xc03fc048c0)
	/home/magik6k/github.com/filecoin-project/go-lotus/chain/messagepool/messagepool.go:1209 +0x1c6
github.com/filecoin-project/lotus/chain/messagepool.(*MessagePool).addLoaded(0xc0423f6d00, {0x64bd058, 0xc041ceff40}, 0xc0426ac090)
	/home/magik6k/github.com/filecoin-project/go-lotus/chain/messagepool/messagepool.go:1058 +0x73
github.com/filecoin-project/lotus/chain/messagepool.(*MessagePool).loadLocal(0xc0423f6d00, {0x64bd058, 0xc041ceff40})
	/home/magik6k/github.com/filecoin-project/go-lotus/chain/messagepool/messagepool.go:1687 +0x28e
github.com/filecoin-project/lotus/chain/messagepool.New.func2()
	/home/magik6k/github.com/filecoin-project/go-lotus/chain/messagepool/messagepool.go:448 +0x6b
created by github.com/filecoin-project/lotus/chain/messagepool.New
	/home/magik6k/github.com/filecoin-project/go-lotus/chain/messagepool/messagepool.go:446 +0xc25

There appear to be no other goroutines in store/index.go.. so possibly this cache is somehow broken, and we're just trashing a lot?

@magik6k
Copy link
Contributor Author

magik6k commented Mar 14, 2023

Nvm, this appears to likely be splitstore related

@magik6k magik6k closed this Mar 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@magik6k