ethdb/pebble: add database backend using pebble #24615

jwasinger · 2022-03-31T08:53:00Z

Uses Pebble patched to expose an API for getting the amount of heap memory allocated through CGo: https://github.com/jwasinger/pebble/tree/mem-stats .

Modifies the system/memory/used and system/memory/held gauges to include Pebble's allocation through CGo.

jwasinger · 2022-04-14T09:55:31Z

Charts from a snap sync (LevelDB is orange, Pebble is green):

Pebble disk usage:

311.7G	/datadir/geth/geth/chaindata/ancient
503.6G	/datadir/geth/geth/chaindata
230.3M	/datadir/geth/geth/ethash
670.1M	/datadir/geth/geth/triecache
4.4M	/datadir/geth/geth/nodes
504.5G	/datadir/geth/geth
4.0K	/datadir/geth/keystore
504.5G	/datadir/geth

Leveldb disk usage:

693.2M	/datadir/geth/geth/triecache
230.3M	/datadir/geth/geth/ethash
311.7G	/datadir/geth/geth/chaindata/ancient
559.7G	/datadir/geth/geth/chaindata
3.6M	/datadir/geth/geth/nodes
560.6G	/datadir/geth/geth
4.0K	/datadir/geth/keystore
560.6G	/datadir/geth

holiman · 2022-05-31T08:53:43Z

ethdb/leveldb/leveldb.go

+	if matches, err := filepath.Glob(file + "/OPTIONS*"); len(matches) > 0 || err != nil {
+		if err != nil {
+			panic(err) // only possible if the pattern is malformed


pebble fingerprinting should be in pebble/pebble.go

I removed the db detection code from both leveldb.go/pebble.go and moved it into core/raw/database.go.

fjl · 2022-07-05T12:37:34Z

Update: still waiting for upstream on cockroachdb/pebble#1628

jwasinger · 2022-07-05T13:12:04Z

Actually they have responded cockroachdb/pebble#1628 (review) .

holiman · 2022-08-16T15:42:03Z

@jwasinger would you mind rebasing this?

jwasinger · 2022-08-17T03:32:27Z

@holiman done.

core/rawdb/database.go

ethdb/pebble/pebble.go

holiman · 2022-08-18T12:49:00Z

ethdb/pebble/pebble.go

+		// Per-level options. Options for at least one level must be specified. The
+		// options for the last level are used for all subsequent levels.
+		Levels: []pebble.LevelOptions{
+			{TargetFileSize: 2 * 1024 * 1024, FilterPolicy: bloom.FilterPolicy(10)},


Did we every try with any larger files in pebble?

I didn't. Maybe @rjl493456442 did at some point?

Yes, I did. As far as I remember it's the same as with LevelDB, compaction makes it go boom.

holiman · 2022-08-29T13:09:44Z

Testing this on two bootnodes which have a very hard time syncing

ansible-playbook playbook.yaml -t geth -l bootnode-azure-westus-001,bootnode-azure-koreasouth-001  -e "geth_image=holiman/geth-experimental:latest" -e "geth_datadir_wipe=partial" -e '{"geth_args_custom":["--backingdb=pebble"]}'

and for comparison:

ansible-playbook playbook.yaml -t geth -l bootnode-azure-brazilsouth-001,bootnode-azure-australiaeast-001  -e "geth_image=holiman/geth-experimental:latest" -e "geth_datadir_wipe=partial"

holiman · 2022-08-29T13:37:40Z

Metrics that do not appear to work:
geth.eth/db/chaindata/disk/read.meter
geth.eth/db/chaindata/compact/writedelay/counter.meter

Edi - here's why the disk read meter is always zerot:

		if db.diskReadMeter != nil {
			db.diskReadMeter.Mark(0) // pebble doesn't track non-compaction reads
		}

Is there any point in even having the metric around? Though I suppose we can keep it for a while, to stay in par with leveldb.

holiman · 2022-08-29T14:08:45Z

It's still very early, but it looks like on our 'weak' azure nodes, pebble is performing a lot better, since it's not being killed-by-compaction:

holiman · 2022-08-29T16:22:57Z

The pebble azure nodes are on ~26%

bootnode-azure-westus-001 geth INFO [08-29|16:20:37.679] State sync in progress synced=26.53% state=51.08GiB accounts=49,463,623@11.06GiB slots=193,320,612@38.88GiB codes=206,295@1.14GiB eta=8h46m27.524s
bootnode-azure-koreasouth-001 geth INFO [08-29|16:20:38.621] State sync in progress synced=26.15% state=50.79GiB accounts=48,789,260@10.90GiB slots=192,700,384@38.77GiB codes=203,451@1.12GiB eta=8h56m8.768s

The non-pebble are on ~15%

bootnode-azure-brazilsouth-001 geth INFO [08-29|16:20:37.959] State sync in progress synced=13.69% state=27.49GiB accounts=26,174,418@5.71GiB slots=104,749,928@21.16GiB codes=117,622@632.17MiB eta=19h31m38.490s
bootnode-azure-australiaeast-001 geth INFO [08-29|16:20:46.154] State sync in progress synced=16.99% state=32.69GiB accounts=32,593,521@7.08GiB slots=123,280,732@24.85GiB codes=141,575@772.87MiB eta=15h11m3.934s

holiman · 2022-08-30T07:13:25Z

Pebble-nodes finished the first phase a couple of hours earlier

Aug 30 03:30:56 bootnode-azure-koreasouth-001 geth INFO [08-30|01:30:55.936] State sync in progress synced=100.00% state=197.38GiB accounts=186,306,958@41.69GiB slots=760,255,029@151.77GiB codes=651,153@3.92GiB eta=-2m14.173s
Aug 30 04:48:43 bootnode-azure-westus-001 geth INFO [08-30|02:48:43.208] State sync in progress synced=100.00% state=197.33GiB accounts=186,328,374@41.69GiB slots=760,016,492@151.72GiB codes=651,467@3.92GiB eta=-2m31.628s

Leveldb-nodes:

Aug 30 06:31:50 bootnode-azure-australiaeast-001 geth INFO [08-30|04:31:50.585] State sync in progress synced=100.00% state=197.47GiB accounts=186,182,442@41.70GiB slots=760,605,354@151.84GiB codes=652,393@3.94GiB eta=-2m43.671s
Aug 30 08:38:23 bootnode-azure-brazilsouth-001 geth INFO [08-30|06:38:23.436] State sync in progress synced=100.00% state=197.51GiB accounts=186,552,432@41.70GiB slots=760,745,777@151.87GiB codes=652,051@3.94GiB eta=-1m27.429s

holiman · 2022-08-30T18:03:58Z

RIght, there's this too:

C:\Users\appveyor\go\pkg\mod\github.com\jwasinger\pebble@v0.0.0-20220816202818-d02a56d8d2b9\internal\batchskl\skl.go:310:18: maxNodesSize (untyped int constant 4294967295) overflows int
C:\Users\appveyor\go\pkg\mod\github.com\jwasinger\pebble@v0.0.0-20220816202818-d02a56d8d2b9\internal\batchskl\skl.go:320:16: cannot use maxNodesSize (untyped int constant 4294967295) as int value in assignment (overflows)

Which afaict would be fixed by cockroachdb/pebble#1619. It has been open since april.

SLoeuillet · 2022-09-16T13:51:57Z

Found a little issue :
Got cache configured to 256K (--cache 262144)
./ethdb/pebble/pebble.go:161: MemTableSize: cache * 1024 * 1024 / 4,

=>

geth[2101456]: Fatal: Failed to register the Ethereum service: MemTableSize (21 G) must be < 4.0 G

So, MemTableSize should be capped to 4GB max

65536 =>
Sep 16 13:51:30 geth01-ethereum-mainnet-eu geth[2105780]: Fatal: Failed to register the Ethereum service: MemTableSize (8.0 G) must be < 4.0 G

32768 =>
Sep 16 13:52:20 geth01-ethereum-mainnet-eu geth[2106436]: Fatal: Failed to register the Ethereum service: MemTableSize (4.0 G) must be < 4.0 G

from https://github.com/cockroachdb/pebble/blob/master/options.go
MemTableSize is an int

SLoeuillet · 2022-09-19T07:33:33Z

On an archive node, it doesn't seem to make sync faster, but at least, the ugly long compaction times (with some going up to 11 days full compacting in a row) are gone

2 archive nodes with standard geth with stair-case effect :

One of those 2 nodes using the pebble branch :

A mix of those 2 with both axes visible :

On 2022-06-29, started 2 archive nodes with geth 1.10.1x then 1.10.2x then 1.11.0
On 2022-07-13, stopped one of those 2, replaced it with 1.11.0, ex_pebble branch after wiping storage out.

holiman · 2022-09-19T07:47:21Z

@SLoeuillet thanks for the feedback and charts! ~~Unfortunately, the Y-axis got a bit cropped out, so I couldn't really figure out how the two charts compared.~~
Would love to see some more charts after a few more days of progress!

holiman · 2022-09-19T07:51:37Z

Ah, the max memtable size is not so much becaues the field is an int (int64), but
https://github.com/cockroachdb/pebble/blob/master/open.go#L38:

	// The max memtable size is limited by the uint32 offsets stored in
	// internal/arenaskl.node, DeferredBatchOp, and flushableBatchEntry.
	maxMemTableSize = 4 << 30 // 4 GB

Co-authored-by: Gary Rong <garyrong0905@gmail.com> foo update

node: fix ddir lookup mistake accounts/abi/bind: fix go.mod replacement for generated binding deps: update pebble + with fix 32-bit build

SLoeuillet · 2022-09-23T14:03:21Z

With a great pleasure, I can announce that my archive node running leveldb standard storage, did just finish to sync to HEAD
2022-06-09 => 2022-09-21

Pebble 1.11.0 based one, started sync on 2022-09-16, currently at 7925855 blocks.

MariusVanDerWijden · 2022-11-09T16:19:30Z

I ran a successful snap sync with it again. Took a pretty long time on a very underprovisioned node (5.8GB usable RAM) but it finished after ~70 hours

holiman · 2023-01-17T13:49:29Z

Triage discussion: I'll take this PR and try to separate the 64-bit and 32-bit, and make it so that we avoid pebble when building 32-bit.

holiman · 2023-01-20T21:10:08Z

Closing in favour of #26517

fjl mentioned this pull request Apr 8, 2022

ethdb: add pebble for experiment #21575

Closed

jwasinger marked this pull request as ready for review May 6, 2022 03:18

jwasinger requested review from fjl, karalabe, holiman and rjl493456442 as code owners May 6, 2022 03:18

jwasinger force-pushed the exp-pebble branch 2 times, most recently from 8df05d4 to f9d8001 Compare May 11, 2022 06:52

jwasinger changed the title ~~ethdb: add pebble for experiment (updated)~~ ethdb: add pebble May 11, 2022

jwasinger force-pushed the exp-pebble branch from 2967ddd to b82ef33 Compare May 31, 2022 07:03

jwasinger added the status:triage label May 31, 2022

holiman reviewed May 31, 2022

View reviewed changes

jwasinger force-pushed the exp-pebble branch from dab5581 to 52fa8af Compare June 1, 2022 01:30

jwasinger force-pushed the exp-pebble branch 2 times, most recently from 2393fd3 to ef8979b Compare June 16, 2022 08:39

fjl changed the title ~~ethdb: add pebble~~ ethdb/pebble: add database backend using pebble Jul 5, 2022

fjl removed the status:triage label Jul 5, 2022

jwasinger force-pushed the exp-pebble branch from ef8979b to 3239203 Compare August 17, 2022 03:31

holiman reviewed Aug 18, 2022

View reviewed changes

core/rawdb/database.go Outdated Show resolved Hide resolved

holiman reviewed Aug 18, 2022

View reviewed changes

core/rawdb/database.go Outdated Show resolved Hide resolved

holiman reviewed Aug 18, 2022

View reviewed changes

holiman force-pushed the exp-pebble branch from 5c98247 to 3a3eb0c Compare August 24, 2022 07:01

holiman added the status:triage label Aug 30, 2022

ligi removed the status:triage label Aug 30, 2022

holiman requested review from gballet and MariusVanDerWijden as code owners August 30, 2022 18:01

holiman force-pushed the exp-pebble branch from 7e883ff to 3a41cd0 Compare September 2, 2022 12:56

jwasinger and others added 3 commits September 19, 2022 10:14

ethdb: use pebble

ffc5e19

Co-authored-by: Gary Rong <garyrong0905@gmail.com> foo update

apply suggested changes

7093bdd

flags: go format

9ff8363

node: fix ddir lookup mistake accounts/abi/bind: fix go.mod replacement for generated binding deps: update pebble + with fix 32-bit build

holiman force-pushed the exp-pebble branch from 3a41cd0 to 9ff8363 Compare September 19, 2022 08:16

ethdb/pebble: respect max memtable size

42380a0

holiman mentioned this pull request Sep 21, 2022

Geth refuses to sync after being a few months offline #25730

Closed

holiman self-assigned this Jan 17, 2023

holiman mentioned this pull request Jan 18, 2023

ethdb: pebble backend (64bit platforms only) #26517

Merged

holiman closed this Jan 20, 2023

quickchase mentioned this pull request Feb 28, 2023

MemTableSize with Pebble #26775

Closed

melekes mentioned this pull request Jan 8, 2024

Add support for pebbleDB cometbft/cometbft-db#90

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ethdb/pebble: add database backend using pebble #24615

ethdb/pebble: add database backend using pebble #24615

jwasinger commented Mar 31, 2022

jwasinger commented Apr 14, 2022

holiman May 31, 2022

jwasinger Jun 1, 2022

fjl commented Jul 5, 2022

jwasinger commented Jul 5, 2022

holiman commented Aug 16, 2022

jwasinger commented Aug 17, 2022

holiman Aug 18, 2022

jwasinger Aug 19, 2022

karalabe Aug 19, 2022

holiman commented Aug 29, 2022

holiman commented Aug 29, 2022 •

edited

Loading

holiman commented Aug 29, 2022

holiman commented Aug 29, 2022 •

edited

Loading

holiman commented Aug 30, 2022

holiman commented Aug 30, 2022

SLoeuillet commented Sep 16, 2022 •

edited

Loading

SLoeuillet commented Sep 19, 2022 •

edited

Loading

holiman commented Sep 19, 2022 •

edited

Loading

holiman commented Sep 19, 2022

SLoeuillet commented Sep 23, 2022

MariusVanDerWijden commented Nov 9, 2022

holiman commented Jan 17, 2023

holiman commented Jan 20, 2023

ethdb/pebble: add database backend using pebble #24615

ethdb/pebble: add database backend using pebble #24615

Conversation

jwasinger commented Mar 31, 2022

jwasinger commented Apr 14, 2022

holiman May 31, 2022

Choose a reason for hiding this comment

jwasinger Jun 1, 2022

Choose a reason for hiding this comment

fjl commented Jul 5, 2022

jwasinger commented Jul 5, 2022

holiman commented Aug 16, 2022

jwasinger commented Aug 17, 2022

holiman Aug 18, 2022

Choose a reason for hiding this comment

jwasinger Aug 19, 2022

Choose a reason for hiding this comment

karalabe Aug 19, 2022

Choose a reason for hiding this comment

holiman commented Aug 29, 2022

holiman commented Aug 29, 2022 • edited Loading

holiman commented Aug 29, 2022

holiman commented Aug 29, 2022 • edited Loading

holiman commented Aug 30, 2022

holiman commented Aug 30, 2022

SLoeuillet commented Sep 16, 2022 • edited Loading

SLoeuillet commented Sep 19, 2022 • edited Loading

holiman commented Sep 19, 2022 • edited Loading

holiman commented Sep 19, 2022

SLoeuillet commented Sep 23, 2022

MariusVanDerWijden commented Nov 9, 2022

holiman commented Jan 17, 2023

holiman commented Jan 20, 2023

holiman commented Aug 29, 2022 •

edited

Loading

holiman commented Aug 29, 2022 •

edited

Loading

SLoeuillet commented Sep 16, 2022 •

edited

Loading

SLoeuillet commented Sep 19, 2022 •

edited

Loading

holiman commented Sep 19, 2022 •

edited

Loading