Add freezer DB debugging tools

[michaelsproul]

Issue Addressed

Related to:

• Fix bug in freezer DB storage of randao_mixes #3011
• Missing chunk in forwards iterator #3455
• State Reconstruction reaches a faulty state #3433

Proposed Changes

This PR adds some extra capabilities to lighthouse db inspect for examining freezer databases.

The main one that's useful for detecting corruption is the --output gaps option which can be used to search for gaps in the linear arrays of block roots, state roots, etc:

./lighthouse db inspect --datadir /mnt/lighthouse/lighthouse/ --freezer --column bbr --output gaps
Aug 26 07:27:34.751 INFO Running database manager for mainnet network
Aug 26 07:27:39.232 INFO Hot-Cold DB initialized                 split_state: 0xd415ec08990df5be242cf0e2c7f32ee8b7419edb1c214c33eb71eaa1eaad99e7, split_slot: 4556160
No gaps found!
Num keys: 35586
Total: 145756192 bytes

The 4 columns of interest are:

• bbr: block roots. Should be gap-free whenever historic block sync has completed. Should contain 1 gap otherwise.
• bsr: state roots. Should be gap-free whenever state reconstruction has completed. Should contain 1 gap otherwise.
• bhr: historic roots. Same gap-status as state roots.
• brm: randao mixes. Same gap-status as state roots.

On my node with historic states all of these 4 columns are gap-free as they should be, indicating that my database (probably) isn't corrupt. Next week or in a follow-up PR I'll add some tools for verifying the actual data stored.

[michaelsproul]

It would be great if people who've been playing around with state reconstruction and noticing corruption issues could try this out CC: @xrchz @JustinZal. I'd be very interested to see what the gap-checker thinks of your databases (you need to stop Lighthouse while running these checks).

If you do find irregular gaps, and I suspect you will, you can try looking at them with the --output values mode. You can fiddle with the --skip and --limit options to reduce the output to a manageable size (I wouldn't recommend running with --output values otherwise).

E.g. on my partly synced node I can do:

$ lighthouse --network mainnet --datadir /media/michael/silver/lh/v3 db inspect --freezer --column bbr --output gaps
Aug 26 07:39:28.665 INFO Running database manager for mainnet network
Aug 26 07:39:28.763 INFO Hot-Cold DB initialized                 split_state: 0xf3abdf015aaf26a68e131313e208955186b89632eb75078f2d3bb34c2756ee09, split_slot: 4556224
gap between keys 1 and 17733 (offset: 2)
Num keys: 17855
Total: 73130016 bytes

and then see the partly filled in chunk with:

$ lighthouse --network mainnet --datadir /media/michael/silver/lh/v3 db inspect --freezer --column bbr --output values --skip 2 --limit 1 
Aug 26 07:40:13.053 INFO Running database manager for mainnet network
Aug 26 07:40:13.148 INFO Hot-Cold DB initialized                 split_state: 0xf3abdf015aaf26a68e131313e208955186b89632eb75078f2d3bb34c2756ee09, split_slot: 4556224
0000000000004545: 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000007b457a43b4c33347792a5d70205a0581c86bbcd59db17d7bbe7dc20d70bff2097b457a43b4c33347792a5d70205a0581c86bbcd59db17d7bbe7dc20d70bff209fb18276397f9fb6f7d84ebd52a4a902ff81a504859171c80e6567aac8e2d8723e737856e5af364cb1884e2539e5f1660c7ec143d444f09ff779c1b6f780d01d9bd0297b7d63e84553c2dac7750915cd7787a0c0724af431f533a35193224d3bc8bef9d98973aa4fffc44921af9469bf97572e51c53dda82c0117787dc09efc531a55865771d319be192df62b8adf09a5c8da83c2f594b7646cc8d1554e4a9f010f19065a937093680640312c46a908782774185fce5b004bd2e1e01ae476149df100c174734338efea7d8a0fe85d40a419ea9a84fb5b3ab000dfbe838877077ab6d879ab7f09499d502eef78069965226e242920f97c12596b560f49ec7392de859d02f2e019bc571a6ddd2791d573aabb34f3016dc5db5517c720488bdc1d4be10077733ae33cc5bd0a2459b79858daac3c3059e270a9c6a8aafec8faa4321939d2d09507f9c97bbfab12291e975e0fad67132ae2987bfa494ab557b18fccbcca5c81de9204c4dda3e1e7146436e3386cb2247fed1dcbd986633eae09347c338423fa9d859b124f1f117eeda618447460a2babc50c1578919b973a2eeed64d11ac576b2071ae811c3af4dc945d3ec4564ed03f270897ea082a4de87be791d68b049a0628c1dc5e4a7c550d2d2be7c9f3a9f91f01bf082e9a3b40c29dc85b917afe34b3fa1b51e3a8f0e2f381ebca658703139d79ab26b093902e7889a5baf0fa8bc012a377a6eacc6eecc0ef191aaf5ff8d106f84b26af04791945d87bbe455b324793ca6845a6cf13c4d61c44694fb7ee60e34a739856034f9755ffe0888bb7f3ffbaad4410390b3897a8378271309b91092823928481e18b4660592f7ff131bf952b027ea45b50f90ffdf8541bce6ab26dbdffd3bf8548a4430a293b186f44a4bb7d3dcdc75c1469bbba59a8c75737d735706d9c66030f7c707d3db598cc404d12944f5a33f604672bedfe88095ecefc4976ebb786f8775b5fc159711f1bee42ad4de3825389a40b8938d3f39a36c931778679b1765e2ed97267d749a5909eb988bd616250d3027bbdd1ad810590d8fe400b21f021411caed6339fcf79d0a1169e026824b53f973153cfb1d4241233e51429c935cbb7f15be42163269df52823b120a2629fdc374395976cff1011e307180ca7fe944934470c0dd571cc544bc7dd965e1e17e2f2d5c55ac09ad88e4e408b701a7debd6d75a8b04cb49de2ec48581a5e41402b3032363a9fffcb5825f3d57be7b17dd07dbf5dbf1ac14e0e19b803cd9bfb79fbb87ff8f2d202d51ed8b09ec7c5d6710787877318f1ee818b27b05579722ca854d4440368e6d58ade1efebfb62f710b7b61cf756a829b25a8bda77744d51cbf1da7445ddc077551a6e91c35ad52d6ba8e950a7d05e4459c48e64968b5b8ad6dea931b0e92eb9b320653838ac8fe154f2bd930723c8021180a1af2fe43b7c68574613d65ce767d00663274bd4c05c588c11c99f129b22e01e1726d3db9a29da269cc7fb8dd46875335a8eb18ff5be6766f9ae75963f9607fac59c6d50ba9860f94ef63a82a36dd734d32a3900a09ffba1252d5313637f07b7da0470fd6b6dbf5474595ec56e50f02577fe89056a69356fbfe39d9a7861455c961329708b8997926d4c78b0be51a20e2898b848441c08770e78377be1985a2b132e69affcbdc9a085cee9a53fc878ec1b3b15e2bc3ce06fcb0b94efe035c8cdcf1326cbf2f2b355def5166460fd7d8e71c764c49a1031258c3297d7c6e6e74af917b9ed985a71b1f99cffdd2410b0c615223eeeb1acb9f1d46f15cecf94b896e2724b7271dd1c0582a2603c44fe99cbb831e2ded1fbead6cb11fc89b4c78dcb8e929ebbe5d30a7b4dcb3a697047f73e955d619fabd98d002ec1a48c4b7fe460e6fc42ac442ae12f4d06def038e8ded209d122b52a7ae184b7bd26442c42ba50e8afed1967496164b7ab451e43e4fa750d50f07bfdd151e1afcf9fe4abfdc6f961eb4eb062d6ee8753c0fff493658a68198ad2a840ee5b39ff7093b4bdb882d530dc64db9218028da2feeb6c63a8c2ad4b800a31884e295683a3199b315e8a314ff3dbb312a5b3f0282d52b6829c983f7e711559c5b13d546212509a9b954225e8ba3c9f48d774a7fc7ce1dd7c76c9e5fc6942e4dc5b2f7e8167612584b8cc0eb0db633465c1bd6e96b8841d26a2e1b682351d4ea5cf13db62dc50a0eaefcda4f56de6b83c39caf7c1345290669cac0c10c14335574d0136e60a183e361b38983025e3e3efce79f7675aba11022604aaf340b89e0e5861fad8f932026fcb39d5c395ebd3e454252cd28dc9ee7006f81e4f5542cf9088c690198755e379cac50a58c4ee50a158c7bdc6ba233489bba74b1154cb37fc6378edb2fa5464cfa7c530df79bf0172e2a4cc5e6ba4311f6dfc7f653adae808c96fe035a600850e7c5533685651966b08392187e9619eb5d95ab7a235d3ff4d11428cffcb2e680b90a39e66dc8cbbab2799c75cad9358b17f407d76db702f18fca3e094d5d9450b286fe0865400f81e480acb038f4a143fdff1f8eec45063662f45f7e8fd540bc24c1472e5d6847d882c820365a92419dcc0f7edb819b1be2a28797c6b49add4823e19547c7105cada9272d12fca0d72037c063520d79eb723be928deae1bdf5c13af9ec7c1832bf6c91f2bb30595a959ea4dcfc5fc1ecdc07820e4502acbfdc5dd221d6d7011e748fb2a01184beab24277fc6aa4af6a61916b7c3a4d9d009cf20056e03769
Num keys: 1
Total: 4096 bytes

[michaelsproul]

Rolling this into tree-states as well.