Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Full Sync #631

Closed
faddat opened this issue Nov 29, 2021 · 7 comments
Closed

Full Sync #631

faddat opened this issue Nov 29, 2021 · 7 comments

Comments

@faddat
Copy link
Member

faddat commented Nov 29, 2021 •
edited
Loading

The goal of this issue is to produce foolproof directions that result in an osmosis archive node that's been synced from genesis to tip.

Syncing using badgerdb and v3.0.0 failed at block: 563470

 osmosisd start --pruning nothing --db_backend badgerdb --p2p.seeds 1b077d96ceeba7ef503fb048f343a538b2dcdf1b@136.243.218.244:26656,2308bed9e096a8b96d2aa343acc1147813c59ed2@3.225.38.25:26656,902bdfe51b6a97cc9369664a21c87ed61d471d2a@136.243.218.243:26656,085f62d67bbf9c501e8ac84d4533440a1eef6c45@95.217.196.54:26656,f515a8599b40f0e84dfad935ba414674ab11a668@osmosis.blockpane.com:26656,63aba59a7da5197c0fbcdc13e760d7561791dca8@162.55.132.230:2000
4:03PM INF starting ABCI with Tendermint
4:03PM INF Starting multiAppConn service impl=multiAppConn module=proxy
4:03PM INF Starting localClient service connection=query impl=localClient module=abci-client
4:03PM INF Starting localClient service connection=snapshot impl=localClient module=abci-client
4:03PM INF Starting localClient service connection=mempool impl=localClient module=abci-client
4:03PM INF Starting localClient service connection=consensus impl=localClient module=abci-client
4:03PM INF Starting EventBus service impl=EventBus module=events
4:03PM INF Starting PubSub service impl=PubSub module=pubsub
4:03PM INF Starting IndexerService service impl=IndexerService module=txindex
4:03PM INF ABCI Handshake App Info hash="2�9��z\x1fmhK��\vE\"���>$qkIb\"�����~�" height=563470 module=consensus protocol-version=0 software-version=
4:03PM INF ABCI Replay Blocks appHeight=563470 module=consensus stateHeight=563470 storeHeight=563471
4:03PM INF Replay last block using real app module=consensus
panic: runtime error: index out of range [7] with length 6

goroutine 1 [running]:
encoding/binary.bigEndian.Uint64(...)
        /usr/lib/go/src/encoding/binary/binary.go:125
github.com/cosmos/cosmos-sdk/types.BigEndianToUint64(...)
        /root/go/pkg/mod/github.com/osmosis-labs/cosmos-sdk@v0.42.10-0.20211028213231-ce6f14aaa6a0/types/utils.go:66
github.com/osmosis-labs/osmosis/x/lockup/keeper.Keeper.getLocksFromIterator({{0x21b2118, 0xc000d08390}, {0x214c810, 0xc000d09f80}, {0x214e688, 0x3096140}, {{0x214c810, 0xc000d09ea0}, {0x21a7d58, 0xc000d08390}, ...}, ...}, ...)
        /root/osmosis/x/lockup/keeper/lock.go:19 +0x35e
github.com/osmosis-labs/osmosis/x/lockup/keeper.Keeper.unlockFromIterator({{0x21b2118, 0xc000d08390}, {0x214c810, 0xc000d09f80}, {0x214e688, 0x3096140}, {{0x214c810, 0xc000d09ea0}, {0x21a7d58, 0xc000d08390}, ...}, ...}, ...)
        /root/osmosis/x/lockup/keeper/lock.go:77 +0xb8
github.com/osmosis-labs/osmosis/x/lockup/keeper.Keeper.WithdrawAllMaturedLocks({{0x21b2118, 0xc000d08390}, {0x214c810, 0xc000d09f80}, {0x214e688, 0x3096140}, {{0x214c810, 0xc000d09ea0}, {0x21a7d58, 0xc000d08390}, ...}, ...}, ...)
        /root/osmosis/x/lockup/keeper/lock.go:46 +0x178
github.com/osmosis-labs/osmosis/x/lockup.EndBlocker({{0x2180928, 0xc0000400b0}, {0x21a5b98, 0xc176fbc640}, {{0xb, 0x1}, {0xc176f61750, 0x9}, 0x8990f, {0x2fe0cf78, ...}, ...}, ...}, ...)
        /root/osmosis/x/lockup/abci.go:22 +0xf8
github.com/osmosis-labs/osmosis/x/lockup.AppModule.EndBlock(...)
        /root/osmosis/x/lockup/module.go:173
github.com/cosmos/cosmos-sdk/types/module.(*Manager).EndBlock(_, {{0x2180928, 0xc0000400b0}, {0x21a5b98, 0xc176fbc640}, {{0xb, 0x1}, {0xc176f61750, 0x9}, 0x8990f, ...}, ...}, ...)
        /root/go/pkg/mod/github.com/osmosis-labs/cosmos-sdk@v0.42.10-0.20211028213231-ce6f14aaa6a0/types/module/module.go:357 +0x33d
github.com/osmosis-labs/osmosis/app.(*OsmosisApp).EndBlocker(...)
        /root/osmosis/app/app.go:594
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).EndBlock(0xc001361860, {0x100000000000020})
        /root/go/pkg/mod/github.com/osmosis-labs/cosmos-sdk@v0.42.10-0.20211028213231-ce6f14aaa6a0/baseapp/abci.go:208 +0x25d
github.com/tendermint/tendermint/abci/client.(*localClient).EndBlockSync(0xc0409b6540, {0xc0409b6540})
        /root/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/abci/client/local_client.go:288 +0xdf
github.com/tendermint/tendermint/proxy.(*appConnConsensus).EndBlockSync(0xc00377da60, {0x20})
        /root/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/proxy/app_conn.go:89 +0x24
github.com/tendermint/tendermint/state.execBlockOnProxyApp({0x2181568, 0xc176f66180}, {0x2195c30, 0xc0409b4b60}, 0xc00000cf00, {0x21a5e08, 0xc0087e0b90}, 0x8990e)
        /root/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/state/execution.go:327 +0x54e
github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x1}, {0xc0409b0358, 0x7}}, {0xc0409b0380, 0x9}, 0x1, 0x8990e, {{0xc0409bc060, ...}, ...}, ...}, ...)
        /root/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/state/execution.go:140 +0x171
github.com/tendermint/tendermint/consensus.(*Handshaker).replayBlock(_, {{{0xb, 0x1}, {0xc0409b0358, 0x7}}, {0xc0409b0380, 0x9}, 0x1, 0x8990e, {{0xc0409bc060, ...}, ...}, ...}, ...)
        /root/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/consensus/replay.go:503 +0x23c
github.com/tendermint/tendermint/consensus.(*Handshaker).ReplayBlocks(_, {{{0xb, 0x1}, {0xc0409b0358, 0x7}}, {0xc0409b0380, 0x9}, 0x1, 0x8990e, {{0xc0409bc060, ...}, ...}, ...}, ...)
        /root/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/consensus/replay.go:416 +0x7ae
github.com/tendermint/tendermint/consensus.(*Handshaker).Handshake(0xc176dcc318, {0x21ae0d0, 0xc01c3ac000})
        /root/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/consensus/replay.go:268 +0x3c8
github.com/tendermint/tendermint/node.doHandshake({_, _}, {{{0xb, 0x1}, {0xc0409b0358, 0x7}}, {0xc0409b0380, 0x9}, 0x1, 0x8990e, ...}, ...)
        /root/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/node/node.go:325 +0x1b8
github.com/tendermint/tendermint/node.NewNode(0xc000e9d2c0, {0x215c710, 0xc00056e280}, 0xc004390a00, {0x213e6a0, 0xc0084117b8}, 0x48bda7, 0x8, 0xc004390c00, {0x2181568, ...}, ...)
        /root/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/node/node.go:733 +0x578
github.com/cosmos/cosmos-sdk/server.startInProcess(_, {{0x0, 0x0, 0x0}, {0x21bd458, 0xc000fce9f0}, {0x0, 0x0}, {0x2189270, 0xc000e49660}, ...}, ...)
        /root/go/pkg/mod/github.com/osmosis-labs/cosmos-sdk@v0.42.10-0.20211028213231-ce6f14aaa6a0/server/start.go:244 +0x625
github.com/cosmos/cosmos-sdk/server.StartCmd.func2(0xc000ef0500, {0xc000f6d1a0, 0x0, 0x6})
        /root/go/pkg/mod/github.com/osmosis-labs/cosmos-sdk@v0.42.10-0.20211028213231-ce6f14aaa6a0/server/start.go:120 +0x193
github.com/spf13/cobra.(*Command).execute(0xc000ef0500, {0xc000f6d0e0, 0x6, 0x6})
        /root/go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:852 +0x60e
github.com/spf13/cobra.(*Command).ExecuteC(0xc000e76500)
        /root/go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:960 +0x3ad
github.com/spf13/cobra.(*Command).Execute(...)
        /root/go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:897
github.com/spf13/cobra.(*Command).ExecuteContext(...)
        /root/go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:890
github.com/osmosis-labs/osmosis/cmd/osmosisd/cmd.Execute(0xc0000001a0)
        /root/osmosis/cmd/osmosisd/cmd/root.go:92 +0xf4
main.main()
        /root/osmosis/cmd/osmosisd/main.go:13 +0x25

I've reverted to v2.0.0 and I'm trying again so that we can test badgerdb against goleveldb.
@faddat
Copy link
Member Author

faddat commented Nov 29, 2021

related to #519

@daniel-farina
Copy link
Contributor

Thank you @faddat for working on this.

Once this is complete. What would be the next steps or outcome of it?
Is it instructions on how to create an archive node that is synced from genesis to date or to actually create a server that runs this archive node?

Thank you I would love to document and learn about this process.

@faddat faddat mentioned this issue Dec 2, 2021
Closed
4 tasks
@faddat
Copy link
Member Author

faddat commented Dec 2, 2021

The outcome is a #627

v2.0.0 dies at the same place, too. back to the initial release, then.

1:33PM INF indexed block height=568469 module=txindex

@daniel-farina daniel-farina added this to the 2021 - December Milestone milestone Dec 2, 2021
@ValarDragon
Copy link
Member

ValarDragon commented Dec 3, 2021 •
edited
Loading

v2 and v3.0.0 should never be used, should be v3.1.0

Should we delete these tags?

@faddat
Copy link
Member Author

faddat commented Dec 4, 2021

oh, you know, perhaps we should. Let me start again

@faddat
Copy link
Member Author

faddat commented Dec 8, 2021

OK, I now have:

  • goleveldb archive node on main relayer
  • badgerdb archive node nearly fully synced
  • rocksdb archive node syncing but still before the upgrade.

@ValarDragon
Copy link
Member

Oh sweet, thanks for checking out the other db's! Sounds like the full sync is working

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
Osmosis Chain Development
Archived in project
Milestone
No milestone
Development

No branches or pull requests
3 participants
@ValarDragon @faddat @daniel-farina