[PoC] Snapshot to bintrie #12

gballet · 2020-03-23T15:20:35Z

Quick and Dirty prototype to build a binary trie from the snapshot, aimed at producing initial data for [1]. It doesn't do any caching, it doesn't do any parallelism, doesn't try to save memory and it stores branches very inefficiently.

@holiman it's also based on a pre-rebase version of trie_gen, I'll rebase if it helps/makes sense.

Refs

https://ethresear.ch/t/overlay-method-for-hex-bin-tree-conversion/7104/3

TODO

Running

This adds a bintrie subcommand to geth that takes the current snapshot and performs the conversion based on that, so conversion can be started with:

geth --snapshot bintrie

holiman · 2020-03-23T17:01:53Z

Yes, please rebase this monster :)

holiman

A few comments (yeah I know it's work in progress and you haven't optimized yet, but I was eager to take a peek)

trie/binary.go

holiman · 2020-03-23T20:04:14Z

trie/binary.go

+	data[0] = byte(len(payload[0]))
+	copy(data[1:], payload[0])
+	data[len(payload[0])+1] = byte(len(payload[1]))
+	copy(data[2+len(payload[0]):], payload[1])
+	data[len(payload[0])+len(payload[1])+2] = byte(len(payload[2]))
+	copy(data[2+len(payload[0])+len(payload[1]):], payload[2])


Where is this format defined?

It's a pure flight of fancy for the moment, I wanted to see what it would take to convert the trie

I changed the format to be closer to what the hex prefix does.

holiman · 2020-03-23T20:08:06Z

trie/binary.go

+	return err
+}
+
+func (t *BinaryTrie) insert(depth int, key, value []byte) error {


Random thought: I wonder if an iterative insert would be faster than a recursive one. I wouldn't mind giving it a go, once we have some benchmark-tests for it

trie/binary.go

trie/stacktrie.go

Co-Authored-By: Martin Holst Swende <martin@swende.se>

holiman · 2020-04-01T12:32:09Z

Do you want me to merge this or want to tick those boxes first?

gballet · 2020-04-01T13:26:02Z

It would be better to tick the boxes first. It shouldn't make it to mainnet until I fixed the OOM that just happened on mon06 😁

…aryTrie

gballet · 2020-04-21T14:24:56Z

After using node extensions:

INFO [04-21|11:04:47.625] Loaded most recent local fast block      number=9689125 hash=d07200��199482 td=14582575985188855549455 age=1mo4d22h
INFO [04-21|11:04:47.626] Generating binary trie                   root=431f3f��6e8372
INFO [04-21|11:04:47.626] Allocated cache and file handles         database=/datadrive/geth/bintrie                cache=128.00MiB handles=1024
INFO [04-21|11:33:17.462] Inserted all leaves                      count=80438713
INFO [04-21|11:33:17.463] Done writing nodes to the DB             count=241316137
INFO [04-21|11:33:17.463] Calculated binary hash                   hash=0x28b39d8423de6c1b8bcf80a3fb66b2139b931c9a61ee465c024a5f70c496d286
INFO [04-21|11:33:17.463] Generation done                          root=431f3f��6e8372 binary root=28b39d��96d286

That's roughly 30 minutes for the account trie conversion, which is a nice improvement.

holiman

LGTM, but I havent' really dived into the binary trie implementation, to be honest.

holiman · 2020-04-22T13:52:19Z

cmd/geth/chaincmd.go

+		return fmt.Errorf("Could not create iterator for root %x: %v", root, err)
+	}
+	log.Info("Generating binary trie", "root", root)
+	generatedRoot := snapshot.GenerateBinaryTree(ctx.GlobalString(utils.DataDirFlag.Name), it)


Does that resolve itself correctly? Like, what if --goerli is specified, is the GlobalString (utils.DataDir... really correct?

While you're at it above, doing log.Info, maybe add

dbPath := ctx.GlobalString(utils.DataDirFlag.Name) log.Info("Generating binary trie", "root", root, "database", dbPath) generatedRoot := snapshot.GenerateBinaryTree(dbPath, it)

holiman · 2020-04-22T13:54:45Z

core/state/snapshot/hextrie_generator.go

+func GenerateBinaryTree(path string, it AccountIterator) common.Hash {
+	db, err := rawdb.NewLevelDBDatabase(path+"/bintrie", 128, 1024, "")
+	if err != nil {
+		panic(fmt.Sprintf("error opening bintrie db, err=%v", err))


Whoa, please don't . You have a live iterator on the original trie db, please back out carefully and close it nicely

holiman · 2020-04-22T13:56:53Z

core/state/snapshot/hextrie_generator.go

+		defer wg.Done()
+		for kv := range btrie.CommitCh {
+			nodeCount++
+			db.Put(kv.Key, kv.Value)


Might go a bit faster if you use db batches.. probably a whole lot faster, actually, since the data you put in there is pretty tiny

Unless you depend on the commit being done synchronously. If you don't require it to be synchronous, you could do two things:

Do batching here,

Add a buffer to the CommitCh. Sometimes it will lag behind, but other times the db insert may be fast (when it only puts it into batch), so a reasonably sized buffer may be good here.

For some reason the correct value can't be retreived from the DB, this will be investigated later.

gballet · 2021-10-11T13:12:01Z

Verkle trees present a promising approach. Closing.

ASISBusiness · 2022-05-06T13:26:41Z

Quick and Dirty prototype to build a binary trie from the snapshot, aimed at producing initial data for [1]. It doesn't do any caching, it doesn't do any parallelism, doesn't try to save memory and it stores branches very inefficiently.

@holiman it's also based on a pre-rebase version of trie_gen, I'll rebase if it helps/makes sense.

Refs

https://ethresear.ch/t/overlay-method-for-hex-bin-tree-conversion/7104/3

TODO

rebase

pruning

extensions

parallelize generation and db writes

rework storage format

benchmark tests

Running

This adds a bintrie subcommand to geth that takes the current snapshot and performs the conversion based on that, so conversion can be started with:
geth --snapshot bintrie

Configures the genesis and network parameters of the protodanksharding devnet.

Downloads mobile

gballet requested a review from holiman as a code owner March 23, 2020 15:20

holiman reviewed Mar 23, 2020

View reviewed changes

trie: binary trie + snapbin command

2d92754

gballet force-pushed the snapshot-to-bintrie branch from 03c51b1 to 2d92754 Compare April 1, 2020 11:19

gballet and others added 2 commits April 1, 2020 13:23

Update trie/binary.go

93375a1

Co-Authored-By: Martin Holst Swende <martin@swende.se>

Update trie/binary.go

d40815b

Co-Authored-By: Martin Holst Swende <martin@swende.se>

gballet added 12 commits April 1, 2020 18:35

Fix bintrie command

2a1629d

Fix a couple bugs in insert and add a corresponding test

62370f0

Fix tests broken after rebase

60cdf15

fix a fix in tryGet tests

4936cc3

Prune the left subtree when inserting into the right subtree

cbe3620

Benchmarking code for bintrie

b8c27d5

trie: introduce extension nodes in search

96d4e08

trie: cleanup and one more extension read test

15e9a10

trie: Add helper to represent bintries in dot format

e954eb7

trie: extension support in binary trie

0369446

Change the node hash format to something hexary-like

65ff453

trie: offload db writes to goroutine + remove dependency on DB in Bin…

3691a07

…aryTrie

gballet added 2 commits April 22, 2020 11:25

trie: refactor binary node creation

50505ee

trie: store binary trie in datadir

8f092fd

holiman approved these changes Apr 22, 2020

View reviewed changes

gballet added 5 commits May 5, 2020 16:24

bugfixes and helpers needed to read from the bintrie

cf0990f

Code to read back from the binary trie DB

309e0ad

For some reason the correct value can't be retreived from the DB, this will be investigated later.

fix: bit overflow in binary prefix's first byte

b8c749d

fix: move value to left of node if right is inserted

c695344

bug: values were overwritten before it hit the DB

9d88014

gballet added 2 commits June 8, 2020 15:55

Increase count for report on number of read keys

684f5fb

trie: change bintrie structure to keep keys at the bottom

50e6855

gballet closed this Oct 11, 2021

holiman pushed a commit that referenced this pull request Nov 26, 2022

add --eip4844 devnet flag (#12)

769f148

Configures the genesis and network parameters of the protodanksharding devnet.

holiman pushed a commit that referenced this pull request Jul 7, 2023

Merge pull request #12 from ethereum/downloads-mobile

bf20f73

Downloads mobile

holiman pushed a commit that referenced this pull request Mar 18, 2024

Capture balance reason (#12)

5466fa3

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[PoC] Snapshot to bintrie #12

[PoC] Snapshot to bintrie #12

gballet commented Mar 23, 2020 •

edited

Loading

holiman commented Mar 23, 2020 via email

holiman left a comment

holiman Mar 23, 2020

gballet Apr 1, 2020

gballet Apr 21, 2020

holiman Mar 23, 2020

gballet Apr 1, 2020

holiman commented Apr 1, 2020

gballet commented Apr 1, 2020

gballet commented Apr 21, 2020

holiman left a comment

holiman Apr 22, 2020

holiman Apr 22, 2020

holiman Apr 22, 2020

holiman Apr 22, 2020

holiman Apr 22, 2020

gballet commented Oct 11, 2021

ASISBusiness commented May 6, 2022

Refs

TODO

Running

[PoC] Snapshot to bintrie #12

[PoC] Snapshot to bintrie #12

Conversation

gballet commented Mar 23, 2020 • edited Loading

Refs

TODO

Running

holiman commented Mar 23, 2020 via email

holiman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

holiman commented Apr 1, 2020

gballet commented Apr 1, 2020

gballet commented Apr 21, 2020

holiman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gballet commented Oct 11, 2021

ASISBusiness commented May 6, 2022

Refs

TODO

Running

gballet commented Mar 23, 2020 •

edited

Loading