move to native badger blockstore; leverage zero-copy View() to deserialize in-place #4681

raulk · 2020-11-01T13:21:01Z

This PR introduces a native badger blockstore that bypasses the datastore abstraction, thus removing layers of indirection, overhead, and allocs from one of the hottest paths in the system.

The native badger blockstore implements the View method, which allows us to directly access an mmap value to deserialize in-place, thus avoiding unnecessary byte slice allocs and copies. See ipfs/go-ipld-cbor#75 for related change.

All usages of the chain datastore have been updated to use the chain blockstore. Low-level commands like state-prune have been adjusted to operate on badger directly.

TODO/decide

Benchmark/comparison. See gist: https://gist.github.com/raulk/1d2a2856b8a77d06cd4a1ba2b68a8fdc/.
With this change, we can no longer use the "measure datastore" wrapper.
- Was this being used at all?
- If yes, we should add a blockstore metrics wrapper (which is anyway planned).
Ideally kill the "/blocks/" prefix/namespace. This increases the footprint for nothing. Also, do not serialize keys in base32. Unfortunately, both these things require a migration.
- @magik6k proposed introducing a version entry on the blockstore so that the code is ready. We'd introduce a lotus shed tool to rewrite the blockstore. That might work, but there are other strategies that might allow us to do this in-place.
- Keeping off-scope of this issue for now.
~~Future: do we want to separate the state and the blockstores?~~
- @raulk: not now; consider when we introduce the two-tier blockstore.
Implement a CBOR object cache, likely using Ristretto, or Freecache. See logical segregation of blockstores + freecache-cached chain and state blockstores #4771.

Kubuxu · 2020-11-01T16:51:35Z

node/repo/blockstore_opts.go

+	// Reduce this from 64MiB to 16MiB. That means badger will hold on to
+	// 20MiB by default instead of 80MiB. This does not appear to have a
+	// significant performance hit.
+	opts.MaxTableSize = 16 << 20


The issue with this is the number of TableFiles that will be created. Lotus for a while now run with default from badger (64M) instead of 20M because of that.

Our badger2 datastore adapter set this value here:

https://github.com/ipfs/go-ds-badger2/blob/a4499bb2f2431c7c430d8a5ffe65c69339e52a00/datastore.go#L107

But it turns out we take those defaults and then entirely override them here:

lotus/node/repo/fsrepo_ds.go

Line 23 in 6cdbf52

opts.Options = dgbadger.DefaultOptions("").WithTruncate(true).

This also means that we lose other defaults that made sense, e.g. ValueLogLoadingMode = FileIO instead of mmap, because we reverted to badger defaults:

https://pkg.go.dev/github.com/dgraph-io/badger#Options.WithValueLogLoadingMode

This could explain some things...

Restored the MaxTableSize value here: 0b8a21e.

I didn't find the ValueLogLoadingMode = FileIO having any benefit (and the results that suggested it are super old) so unless you have some results confirminig it, I would stay with dgraph/badger defaults.

FileIO sabotages the benefits of View(), so we're back to mmap.

lib/blockstore/badger/badger_test_suite.go

lib/blockstore/badger/badger.go

…CAR.

Kubuxu

SGWM but lack of support of View from CachingBS and IDWrapperBS will stop us from using the View.

raulk · 2020-11-09T19:29:02Z

@Kubuxu is right, and this is evident in profiles. This PR is incomplete until I add the View() method to the various overlay blockstore concoctions we have.

lib/bufbstore/buf_bstore.go

This reverts commit f8c3756.

Kubuxu · 2020-11-10T19:39:06Z

chain/vm/vm.go

+		bs.chargeGas(bs.pricelist.OnIpldGet())
+		return v.View(c, func(b []byte) error {
+			// we have successfully retrieved the value; charge for it, even if the user-provided function fails.
+			bs.chargeGas(newGasCharge("OnIpldViewEnd", 0, 0).WithExtra(len(b)))


These two are only signaling charges, they have no consensus importantce so it is fine if they are inside the View.

magik6k

Looks good, just 1 comment

chain/vm/vm.go

arajasek

Approved magiks suggestion

chain/vm/vm.go

raulk added 6 commits November 1, 2020 12:55

add a native badger blockstore with View() method.

ce27b13

make the lotus node use the new native badger blockstore.

d2e2322

fixup.

9437136

migrate repo.Datastore(/chain) to repo.Blockstore().

099c4b5

update/add dependencies.

95f512a

Merge branch 'master' into badger-viewable

a16d7f2

raulk requested review from Kubuxu, magik6k and whyrusleeping as code owners November 1, 2020 13:21

raulk marked this pull request as draft November 1, 2020 13:21

fix lotus-shed datastore commands.

d8d8537

Kubuxu reviewed Nov 1, 2020

View reviewed changes

raulk added 6 commits November 1, 2020 16:56

make repo.Blockstore() idempotent; wrap in IDStore.

5a98660

fix lint.

72e573d

badger: restore prev. max table size (64MiB; default).

0b8a21e

go mod tidy.

634467d

write heap and allocs profile in lotus-bench.

c2355b1

lotus-bench: add support for native badger blockstore.

aece624

raulk commented Nov 1, 2020

View reviewed changes

lib/blockstore/badger/badger_test_suite.go Outdated Show resolved Hide resolved

lib/blockstore/badger/badger.go Outdated Show resolved Hide resolved

raulk added 6 commits November 1, 2020 19:52

lotus-bench: improve for observability.

35ccd73

lotus-bench: native badger SyncWrites=false.

44e34d9

actually register prometheus gauge.

f4e13ff

support legacy keying: base32 multihashes *sigh*.

7facdf6

introduce non-pooled StorageKey method.

d1ebf3c

improve lotus-bench; support running validation on a store without a …

842c8ca

…CAR.

raulk force-pushed the badger-viewable branch from 02afe46 to 842c8ca Compare November 3, 2020 17:35

raulk added 3 commits November 3, 2020 18:11

lotus-bench: add ability to specify tipsets.

581ac5b

lotus-bench: make start tipset walkback start from end tipset.

1841812

Merge branch 'master' into badger-viewable

8091113

raulk marked this pull request as ready for review November 3, 2020 21:13

Kubuxu approved these changes Nov 9, 2020

View reviewed changes

arajasek added this to the 🖇Lotus Stabilization (1.1.3) milestone Nov 9, 2020

raulk marked this pull request as draft November 10, 2020 09:09

raulk added 4 commits November 10, 2020 12:40

add the viewable trait to our gazillion blockstores.

54bf7c9

import Viewable non-terminal blockstores.

577476b

remove unused and misleading cachebs store.

6d78de9

migrate direct usages blockstore.{Get=>View}.

af7236f

Kubuxu reviewed Nov 10, 2020

View reviewed changes

lib/bufbstore/buf_bstore.go Outdated Show resolved Hide resolved

raulk added 3 commits November 10, 2020 15:02

fix comment.

360194b

(to be reverted) add debug.PrintStack.

f8c3756

vm: construct CBOR store properly.

38c404e

raulk force-pushed the badger-viewable branch from 0182d1b to 38c404e Compare November 10, 2020 15:49

raulk added 4 commits November 10, 2020 16:08

Revert "(to be reverted) add debug.PrintStack."

339391e

This reverts commit f8c3756.

upgrade dependencies.

379dd02

remove benchmark.

d79ff24

fix comment.

659ceaa

raulk marked this pull request as ready for review November 10, 2020 17:51

raulk requested review from Kubuxu, whyrusleeping and magik6k November 10, 2020 18:17

Kubuxu reviewed Nov 10, 2020

View reviewed changes

Kubuxu approved these changes Nov 10, 2020

View reviewed changes

magik6k reviewed Nov 10, 2020

View reviewed changes

chain/vm/vm.go Outdated Show resolved Hide resolved

arajasek approved these changes Nov 10, 2020

View reviewed changes

chain/vm/vm.go Outdated Show resolved Hide resolved

raulk added 2 commits November 10, 2020 22:34

fix condition.

8f5847b

skip callers in badger log for useful line info.

bc6965c

Kubuxu approved these changes Nov 10, 2020

View reviewed changes

arajasek merged commit 735c04f into master Nov 10, 2020

arajasek deleted the badger-viewable branch November 10, 2020 23:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

move to native badger blockstore; leverage zero-copy View() to deserialize in-place #4681

move to native badger blockstore; leverage zero-copy View() to deserialize in-place #4681

raulk commented Nov 1, 2020 •

edited

Loading

Kubuxu Nov 1, 2020

raulk Nov 1, 2020

raulk Nov 1, 2020

Kubuxu Nov 6, 2020

raulk Nov 10, 2020

Kubuxu left a comment

raulk commented Nov 9, 2020

Kubuxu Nov 10, 2020

magik6k left a comment

arajasek left a comment

move to native badger blockstore; leverage zero-copy View() to deserialize in-place #4681

move to native badger blockstore; leverage zero-copy View() to deserialize in-place #4681

Conversation

raulk commented Nov 1, 2020 • edited Loading

TODO/decide

Kubuxu Nov 1, 2020

Choose a reason for hiding this comment

raulk Nov 1, 2020

Choose a reason for hiding this comment

raulk Nov 1, 2020

Choose a reason for hiding this comment

Kubuxu Nov 6, 2020

Choose a reason for hiding this comment

raulk Nov 10, 2020

Choose a reason for hiding this comment

Kubuxu left a comment

Choose a reason for hiding this comment

raulk commented Nov 9, 2020

Kubuxu Nov 10, 2020

Choose a reason for hiding this comment

magik6k left a comment

Choose a reason for hiding this comment

arajasek left a comment

Choose a reason for hiding this comment

raulk commented Nov 1, 2020 •

edited

Loading