[CAD-3383] Init db-analyser from stored ledger state snapshot #3397
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
EncodePanda
changed the title
EncodePanda
force-pushed
the
EncodePanda/CAD-3383/init-from-snapshot
branch
2 times, most recently
from
2058b15
to
3390718
Compare
nfrisby
requested changes
Sep 28, 2021
nfrisby
reviewed
Sep 28, 2021
EncodePanda
force-pushed
the
EncodePanda/CAD-3383/init-from-snapshot
branch
from
3390718
to
214cf82
Compare
nfrisby
requested changes
Oct 4, 2021
nfrisby
approved these changes
Oct 5, 2021
EncodePanda
force-pushed
the
EncodePanda/CAD-3383/init-from-snapshot
branch
2 times, most recently
from
5427bdf
to
254a8c6
Compare
Move implementation of streamAll to streamFrom, where `from` becomes a parameter to that function. Then streamAll can just call streamFrom with a GenesisPoint as an argument.
EncodePanda
force-pushed
the
EncodePanda/CAD-3383/init-from-snapshot
branch
from
254a8c6
to
1241181
Compare
|
bors merge |
iohk-bors bot
added a commit
that referenced
this pull request
Oct 6, 2021
3397: [CAD-3383] Init db-analyser from stored ledger state snapshot r=EncodePanda a=EncodePanda
## TL;DR
You can now start db-analyser starting from a particular ledger state snapshot by providing `--analyse-from`
Note that this feature is only available when `--only-immutable-db` is active.
## Example usage
Consider ChainDB that holds snapshots `39842331` and `40174168`
```
λ ls -lath db_mainnet_with_alonzo_40217333/ledger
total 2.9G
drwxr-xr-x 6 encodepanda 192 Sep 28 15:35 .
-rw-r--r-- 1 encodepanda 732M Sep 28 13:36 39842331_db-analyser
-rw-r--r-- 1 encodepanda 747M Sep 22 10:22 40174168_db-analyser
(...)
```
We can now run the `db-analyser` starting from one of the snapshots. Here we start from snapshot `39842331`, we ask for 10 blocks only and we want to print out their block and slot numbers
```
λ cabal run ouroboros-consensus-cardano:db-analyser -- \
--db db_mainnet_with_alonzo_40217333 cardano \
--configByron dbacfg/mainnet-byron-genesis.json \
--configShelley dbacfg/mainnet-shelley-genesis.json \
--nonce 1a3be38bcbb7911969283716ad7aa550250226b76a61fc51cc9a9a35d9276d81 \
--configAlonzo dbacfg/mainnet-alonzo-genesis.json \
--show-slot-block-no \
--num-blocks-to-process 10 \
--analyse-from 39842331
--only-immutable-db
BlockNo 6232309 SlotNo 39842333
BlockNo 6232310 SlotNo 39842339
BlockNo 6232311 SlotNo 39842355
BlockNo 6232312 SlotNo 39842379
BlockNo 6232313 SlotNo 39842411
BlockNo 6232314 SlotNo 39842441
BlockNo 6232315 SlotNo 39842449
BlockNo 6232316 SlotNo 39842460
BlockNo 6232317 SlotNo 39842492
BlockNo 6232318 SlotNo 39842508
ImmutableDB tip: At (Block {blockPointSlot = SlotNo 40174168, blockPointHash = db782958bd783e491d750a84080194d118828b6603130a21a4b9d66e855d6032})
```
We can now even store a new snapshot at `39842441` by initializing from the existing snapshot at `39842331`
```
λ cabal run ouroboros-consensus-cardano:db-analyser -- \
--db db_mainnet_with_alonzo_40217333 cardano \
--configByron dbacfg/mainnet-byron-genesis.json \
--configShelley dbacfg/mainnet-shelley-genesis.json \
--nonce 1a3be38bcbb7911969283716ad7aa550250226b76a61fc51cc9a9a35d9276d81 \
--configAlonzo dbacfg/mainnet-alonzo-genesis.json \
--store-ledger 39842441 \
--num-blocks-to-process 10
--only-immutable-db
--analyse-from 39842331
About to store snapshot of a ledger at SlotNo 39842441 this might take a while...
storing state at BlockNo 6232314 SlotNo 39842441 6d793c1dfc019f152d0df9a7eececdf0de0a5982aaf7cdc80555904b36b24ce8
```
And now we can check that it actually worked.
1. new snapshot was created (`39842441_db-analyser`)
```
λ ls -lath db_mainnet_with_alonzo_40217333/ledger
total 3.7G
-rw-r--r-- 1 encodepanda 732M Sep 28 15:43 39842441_db-analyser
drwxr-xr-x 7 encodepanda 224 Sep 28 15:43 .
-rw-r--r-- 1 encodepanda 732M Sep 28 13:36 39842331_db-analyser
-rw-r--r-- 1 encodepanda 747M Sep 22 10:22 40174168_db-analyser
(...)
```
2. we can start db-analyser from that newly created snapshot
```
λ cabal run ouroboros-consensus-cardano:db-analyser -- \
--db db_mainnet_with_alonzo_40217333 cardano \
--configByron dbacfg/mainnet-byron-genesis.json \
--configShelley dbacfg/mainnet-shelley-genesis.json \
--nonce 1a3be38bcbb7911969283716ad7aa550250226b76a61fc51cc9a9a35d9276d81 \
--configAlonzo dbacfg/mainnet-alonzo-genesis.json \
--show-slot-block-no \
--num-blocks-to-process 15 \
--only-immutable-db \
--analyse-from 39842441
BlockNo 6232315 SlotNo 39842449
BlockNo 6232316 SlotNo 39842460
BlockNo 6232317 SlotNo 39842492
BlockNo 6232318 SlotNo 39842508
BlockNo 6232319 SlotNo 39842527
BlockNo 6232320 SlotNo 39842542
BlockNo 6232321 SlotNo 39842545
BlockNo 6232322 SlotNo 39842580
BlockNo 6232323 SlotNo 39842586
BlockNo 6232324 SlotNo 39842602
BlockNo 6232325 SlotNo 39842616
BlockNo 6232326 SlotNo 39842658
BlockNo 6232327 SlotNo 39842695
BlockNo 6232328 SlotNo 39842708
BlockNo 6232329 SlotNo 39842731
ChainDB tip: At (Block {blockPointSlot = SlotNo 40217333, blockPointHash = fd4487549a0d2c6dcba32255c2a4f7e1419cb1c88a07b8872a32db19e4b02a41})
```
Co-authored-by: EncodePanda <paul.szulc@gmail.com>
|
Build failed: |
Note that this flag is only optional if user is running --only-immutable-db. We made that choice, because when running from ChainDB, the DB will reply its ledger state(s) from most recent ledger. This might be especially problematic when doing profiling - user would see obfuscated data because additional objects would be loaded into memory (intermediate ledgers etc). Thus we've decided (at least for now) to not make it an option when running db-analyser with whole ChainDB. With this flag user can select which snapshot will be chosen to initalize the db-analyser with. User provides the slot number and if the snapshot with a name 'slotnumber_db-analyser' exists, the db-analyser tool will start working starting from that snapshot (instead of starting from genesis). This ledger will be then can be used with processAll function (see --store-ledger as an example usage).
EncodePanda
force-pushed
the
EncodePanda/CAD-3383/init-from-snapshot
branch
from
1241181
to
c71eea4
Compare
|
bors merge |
|
Build succeeded: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
TL;DR
You can now start db-analyser starting from a particular ledger state snapshot by providing
--analyse-fromNote that this feature is only available when
--only-immutable-dbis active.Example usage
Consider ChainDB that holds snapshots
39842331and40174168We can now run the
db-analyserstarting from one of the snapshots. Here we start from snapshot39842331, we ask for 10 blocks only and we want to print out their block and slot numbersWe can now even store a new snapshot at
39842441by initializing from the existing snapshot at39842331And now we can check that it actually worked.
39842441_db-analyser)