Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove fast node cache and benchmark #1041

Closed
Tracked by #1016
p0mvn opened this issue Mar 4, 2022 · 6 comments
Closed
Tracked by #1016

Remove fast node cache and benchmark #1041

p0mvn opened this issue Mar 4, 2022 · 6 comments
Assignees
@p0mvn
Labels
feature New feature or request needs-investigation T:task ⚙️ A task belongs to a story

Comments

@p0mvn
Copy link
Member

p0mvn commented Mar 4, 2022 •
edited
Loading

Background

We suspect that fastNodeCache in nodeDb may not be very useful. We should attempt to remove it and benchmark the results

Acceptance Criteria

  • remove fastNodeCache from nodeDb
  • benchmark
  • merge the change depending on whether an improvement is observed
@p0mvn p0mvn mentioned this issue Mar 4, 2022
Closed
14 tasks
@p0mvn p0mvn added needs-investigation T:task ⚙️ A task belongs to a story labels Mar 4, 2022
@p0mvn p0mvn self-assigned this Mar 4, 2022
@p0mvn p0mvn added the feature New feature or request label Mar 4, 2022
@p0mvn
Copy link
Member Author

p0mvn commented Mar 4, 2022

cache.log:

iavl: 
git commit: 
git branch: 
go version go1.17.3 linux/amd64

Init Tree took 66.88 MB
goos: linux
goarch: amd64
pkg: github.com/cosmos/iavl/benchmarks
cpu: AMD Ryzen 7 3700X 8-Core Processor             
BenchmarkMedium/goleveldb-100000-100-16-40/query-no-in-tree-guarantee-fast-16             409921              2952 ns/op             814 B/op      16 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/query-no-in-tree-guarantee-fast-16             413432              3032 ns/op             814 B/op      16 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/query-no-in-tree-guarantee-fast-16             437080              2921 ns/op             814 B/op      16 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/query-no-in-tree-guarantee-fast-16             450187              2984 ns/op             814 B/op      16 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/query-no-in-tree-guarantee-fast-16             415393              3005 ns/op             814 B/op      16 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/query-no-in-tree-guarantee-slow-16             116660              9018 ns/op            1404 B/op      24 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/query-no-in-tree-guarantee-slow-16             126883              9616 ns/op            1412 B/op      24 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/query-no-in-tree-guarantee-slow-16             109087              9539 ns/op            1408 B/op      24 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/query-no-in-tree-guarantee-slow-16             105081              9530 ns/op            1410 B/op      24 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/query-no-in-tree-guarantee-slow-16             107452              9870 ns/op            1422 B/op      25 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/query-hits-fast-16                            2281778               512.9 ns/op            22 B/op       0 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/query-hits-fast-16                            2415718               467.0 ns/op            13 B/op       0 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/query-hits-fast-16                            2850028               409.7 ns/op             9 B/op       0 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/query-hits-fast-16                            2650272               400.4 ns/op             7 B/op       0 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/query-hits-fast-16                            3019638               401.9 ns/op             5 B/op       0 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/query-hits-slow-16                              83739             13331 ns/op            1989 B/op      34 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/query-hits-slow-16                              82207             13414 ns/op            1997 B/op      34 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/query-hits-slow-16                              87763             13213 ns/op            1997 B/op      34 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/query-hits-slow-16                              85336             13063 ns/op            1994 B/op      34 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/query-hits-slow-16                              88747             13510 ns/op            1994 B/op      34 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/iteration-fast-16                                  21          65798802 ns/op        29327116 B/op  522985 allocs/op
--- BENCH: BenchmarkMedium/goleveldb-100000-100-16-40/iteration-fast-16
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
        ... [output truncated]
BenchmarkMedium/goleveldb-100000-100-16-40/iteration-fast-16                                  24          65081808 ns/op        29327085 B/op  522985 allocs/op
--- BENCH: BenchmarkMedium/goleveldb-100000-100-16-40/iteration-fast-16
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
        ... [output truncated]
BenchmarkMedium/goleveldb-100000-100-16-40/iteration-fast-16                                  19          63440390 ns/op        29327105 B/op  522984 allocs/op
--- BENCH: BenchmarkMedium/goleveldb-100000-100-16-40/iteration-fast-16
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
        ... [output truncated]
BenchmarkMedium/goleveldb-100000-100-16-40/iteration-fast-16                                  16          70851619 ns/op        29327165 B/op  522986 allocs/op
--- BENCH: BenchmarkMedium/goleveldb-100000-100-16-40/iteration-fast-16
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
        ... [output truncated]
BenchmarkMedium/goleveldb-100000-100-16-40/iteration-fast-16                                  24          63981203 ns/op        29327077 B/op  522984 allocs/op
--- BENCH: BenchmarkMedium/goleveldb-100000-100-16-40/iteration-fast-16
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
        ... [output truncated]
BenchmarkMedium/goleveldb-100000-100-16-40/iteration-slow-16                                   1        1061595563 ns/op        234506024 B/op        3990789 allocs/op
--- BENCH: BenchmarkMedium/goleveldb-100000-100-16-40/iteration-slow-16
    bench_test.go:143: completed 100000 iterations
BenchmarkMedium/goleveldb-100000-100-16-40/iteration-slow-16                                   1        1302907527 ns/op        276163048 B/op        4705768 allocs/op
--- BENCH: BenchmarkMedium/goleveldb-100000-100-16-40/iteration-slow-16
    bench_test.go:143: completed 100000 iterations
BenchmarkMedium/goleveldb-100000-100-16-40/iteration-slow-16                                   1        1306277937 ns/op        276107952 B/op        4705548 allocs/op
--- BENCH: BenchmarkMedium/goleveldb-100000-100-16-40/iteration-slow-16
    bench_test.go:143: completed 100000 iterations
BenchmarkMedium/goleveldb-100000-100-16-40/iteration-slow-16                                   1        1244302858 ns/op        276114160 B/op        4705554 allocs/op
--- BENCH: BenchmarkMedium/goleveldb-100000-100-16-40/iteration-slow-16
    bench_test.go:143: completed 100000 iterations
BenchmarkMedium/goleveldb-100000-100-16-40/iteration-slow-16                                   1        1290297055 ns/op        276119104 B/op        4705556 allocs/op
--- BENCH: BenchmarkMedium/goleveldb-100000-100-16-40/iteration-slow-16
    bench_test.go:143: completed 100000 iterations
BenchmarkMedium/goleveldb-100000-100-16-40/update-16                                        8560            138764 ns/op           48101 B/op     481 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/update-16                                        7516            182969 ns/op           51551 B/op     545 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/update-16                                        7371            178423 ns/op           51944 B/op     595 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/update-16                                        6032            182461 ns/op           52321 B/op     598 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/update-16                                        6693            180266 ns/op           51656 B/op     592 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/block-16                                           55          22694269 ns/op         6382976 B/op   71897 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/block-16                                           70          22618362 ns/op         6422396 B/op   72678 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/block-16                                           55          24569598 ns/op         6588177 B/op   73777 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/block-16                                           49          22197088 ns/op         6183273 B/op   71091 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/block-16                                           46          23527320 ns/op         6564015 B/op   74866 allocs/op
PASS
ok      github.com/cosmos/iavl/benchmarks       72.810s

nocache.log:

iavl: 
git commit: 
git branch: 
go version go1.17.3 linux/amd64

Init Tree took 48.34 MB
goos: linux
goarch: amd64
pkg: github.com/cosmos/iavl/benchmarks
cpu: AMD Ryzen 7 3700X 8-Core Processor             
BenchmarkMedium/goleveldb-100000-100-16-40/query-no-in-tree-guarantee-fast-16         	  390157	      2903 ns/op	     814 B/op	      16 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/query-no-in-tree-guarantee-fast-16         	  399000	      2963 ns/op	     814 B/op	      16 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/query-no-in-tree-guarantee-fast-16         	  402136	      3179 ns/op	     814 B/op	      16 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/query-no-in-tree-guarantee-fast-16         	  420476	      2937 ns/op	     814 B/op	      16 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/query-no-in-tree-guarantee-fast-16         	  417190	      3076 ns/op	     814 B/op	      16 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/query-no-in-tree-guarantee-slow-16         	  113926	      9762 ns/op	    1409 B/op	      24 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/query-no-in-tree-guarantee-slow-16         	  105357	      9851 ns/op	    1406 B/op	      24 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/query-no-in-tree-guarantee-slow-16         	  109642	      9834 ns/op	    1405 B/op	      24 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/query-no-in-tree-guarantee-slow-16         	  103359	      9763 ns/op	    1406 B/op	      24 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/query-no-in-tree-guarantee-slow-16         	  125224	     10063 ns/op	    1409 B/op	      24 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/query-hits-fast-16                         	  345147	      3447 ns/op	     886 B/op	      16 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/query-hits-fast-16                         	  353240	      3364 ns/op	     886 B/op	      16 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/query-hits-fast-16                         	  350920	      3338 ns/op	     886 B/op	      16 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/query-hits-fast-16                         	  372626	      3307 ns/op	     886 B/op	      16 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/query-hits-fast-16                         	  367660	      3294 ns/op	     886 B/op	      16 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/query-hits-slow-16                         	   86119	     12926 ns/op	    1990 B/op	      34 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/query-hits-slow-16                         	   83398	     12708 ns/op	    1990 B/op	      34 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/query-hits-slow-16                         	   86103	     14252 ns/op	    2003 B/op	      34 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/query-hits-slow-16                         	   90589	     13302 ns/op	    1989 B/op	      34 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/query-hits-slow-16                         	   85428	     13497 ns/op	    2000 B/op	      34 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/iteration-fast-16                          	      21	  68717304 ns/op	29327113 B/op	  522985 allocs/op
--- BENCH: BenchmarkMedium/goleveldb-100000-100-16-40/iteration-fast-16
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
	... [output truncated]
BenchmarkMedium/goleveldb-100000-100-16-40/iteration-fast-16                          	      15	  70264586 ns/op	29327162 B/op	  522985 allocs/op
--- BENCH: BenchmarkMedium/goleveldb-100000-100-16-40/iteration-fast-16
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
	... [output truncated]
BenchmarkMedium/goleveldb-100000-100-16-40/iteration-fast-16                          	      18	  70958463 ns/op	29327138 B/op	  522985 allocs/op
--- BENCH: BenchmarkMedium/goleveldb-100000-100-16-40/iteration-fast-16
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
	... [output truncated]
BenchmarkMedium/goleveldb-100000-100-16-40/iteration-fast-16                          	      15	  72209305 ns/op	29327158 B/op	  522985 allocs/op
--- BENCH: BenchmarkMedium/goleveldb-100000-100-16-40/iteration-fast-16
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
	... [output truncated]
BenchmarkMedium/goleveldb-100000-100-16-40/iteration-fast-16                          	      22	  73218143 ns/op	29327098 B/op	  522985 allocs/op
--- BENCH: BenchmarkMedium/goleveldb-100000-100-16-40/iteration-fast-16
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
    bench_test.go:143: completed 100000 iterations
	... [output truncated]
BenchmarkMedium/goleveldb-100000-100-16-40/iteration-slow-16                          	       1	1103013134 ns/op	234491104 B/op	 3991442 allocs/op
--- BENCH: BenchmarkMedium/goleveldb-100000-100-16-40/iteration-slow-16
    bench_test.go:143: completed 100000 iterations
BenchmarkMedium/goleveldb-100000-100-16-40/iteration-slow-16                          	       1	1259815260 ns/op	276134112 B/op	 4705557 allocs/op
--- BENCH: BenchmarkMedium/goleveldb-100000-100-16-40/iteration-slow-16
    bench_test.go:143: completed 100000 iterations
BenchmarkMedium/goleveldb-100000-100-16-40/iteration-slow-16                          	       1	1268760085 ns/op	276095808 B/op	 4705374 allocs/op
--- BENCH: BenchmarkMedium/goleveldb-100000-100-16-40/iteration-slow-16
    bench_test.go:143: completed 100000 iterations
BenchmarkMedium/goleveldb-100000-100-16-40/iteration-slow-16                          	       1	1308360325 ns/op	276080216 B/op	 4705371 allocs/op
--- BENCH: BenchmarkMedium/goleveldb-100000-100-16-40/iteration-slow-16
    bench_test.go:143: completed 100000 iterations
BenchmarkMedium/goleveldb-100000-100-16-40/iteration-slow-16                          	       1	1276755134 ns/op	276095856 B/op	 4705377 allocs/op
--- BENCH: BenchmarkMedium/goleveldb-100000-100-16-40/iteration-slow-16
    bench_test.go:143: completed 100000 iterations
BenchmarkMedium/goleveldb-100000-100-16-40/update-16                                  	    9696	    146592 ns/op	   47615 B/op	     475 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/update-16                                  	    6853	    181887 ns/op	   51788 B/op	     542 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/update-16                                  	    7207	    181912 ns/op	   54073 B/op	     614 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/update-16                                  	    6890	    183622 ns/op	   54070 B/op	     604 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/update-16                                  	    6342	    194800 ns/op	   55598 B/op	     597 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/block-16                                   	      39	  26758858 ns/op	 7043085 B/op	   77740 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/block-16                                   	      55	  23525001 ns/op	 6802199 B/op	   73821 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/block-16                                   	      46	  23722432 ns/op	 6539204 B/op	   72672 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/block-16                                   	      70	  23866420 ns/op	 6759421 B/op	   74000 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/block-16                                   	      48	  27214536 ns/op	 7338237 B/op	   76913 allocs/op
PASS
ok  	github.com/cosmos/iavl/benchmarks	69.591s

@p0mvn
Copy link
Member Author

p0mvn commented Mar 4, 2022 •
edited
Loading

Comparison:

benchstat cache.log nocache.log
name                                                                  old time/op    new time/op    delta
Medium/goleveldb-100000-100-16-40/query-no-in-tree-guarantee-fast-16    2.98µs ± 2%    3.01µs ± 6%       ~     (p=1.000 n=5+5)
Medium/goleveldb-100000-100-16-40/query-no-in-tree-guarantee-slow-16    9.51µs ± 5%    9.85µs ± 2%       ~     (p=0.095 n=5+5)
Medium/goleveldb-100000-100-16-40/query-hits-fast-16                     438ns ±17%    3350ns ± 3%   +664.18%  (p=0.008 n=5+5)
Medium/goleveldb-100000-100-16-40/query-hits-slow-16                    13.3µs ± 2%    13.3µs ± 7%       ~     (p=0.841 n=5+5)
Medium/goleveldb-100000-100-16-40/iteration-fast-16                     65.8ms ± 8%    71.1ms ± 3%     +7.96%  (p=0.032 n=5+5)
Medium/goleveldb-100000-100-16-40/iteration-slow-16                      1.24s ±14%     1.24s ±11%       ~     (p=1.000 n=5+5)
Medium/goleveldb-100000-100-16-40/update-16                              181µs ± 1%     178µs ±18%       ~     (p=0.730 n=4+5)
Medium/goleveldb-100000-100-16-40/block-16                              23.1ms ± 6%    25.0ms ± 9%       ~     (p=0.095 n=5+5)

name                                                                  old alloc/op   new alloc/op   delta
Medium/goleveldb-100000-100-16-40/query-no-in-tree-guarantee-fast-16      814B ± 0%      814B ± 0%       ~     (all equal)
Medium/goleveldb-100000-100-16-40/query-no-in-tree-guarantee-slow-16    1.41kB ± 1%    1.41kB ± 0%       ~     (p=0.286 n=5+5)
Medium/goleveldb-100000-100-16-40/query-hits-fast-16                     11.2B ±96%    886.0B ± 0%  +7810.71%  (p=0.008 n=5+5)
Medium/goleveldb-100000-100-16-40/query-hits-slow-16                    1.99kB ± 0%    1.99kB ± 0%       ~     (p=1.000 n=5+5)
Medium/goleveldb-100000-100-16-40/iteration-fast-16                     29.3MB ± 0%    29.3MB ± 0%       ~     (p=0.421 n=5+5)
Medium/goleveldb-100000-100-16-40/iteration-slow-16                      276MB ± 0%     276MB ± 0%       ~     (p=0.200 n=4+4)
Medium/goleveldb-100000-100-16-40/update-16                             51.1kB ± 6%    52.6kB ±10%       ~     (p=0.310 n=5+5)
Medium/goleveldb-100000-100-16-40/block-16                              6.43MB ± 4%    6.90MB ± 6%     +7.28%  (p=0.032 n=5+5)

name                                                                  old allocs/op  new allocs/op  delta
Medium/goleveldb-100000-100-16-40/query-no-in-tree-guarantee-fast-16      16.0 ± 0%      16.0 ± 0%       ~     (all equal)
Medium/goleveldb-100000-100-16-40/query-no-in-tree-guarantee-slow-16      24.0 ± 0%      24.0 ± 0%       ~     (all equal)
Medium/goleveldb-100000-100-16-40/query-hits-fast-16                      0.00          16.00 ± 0%      +Inf%  (p=0.008 n=5+5)
Medium/goleveldb-100000-100-16-40/query-hits-slow-16                      34.0 ± 0%      34.0 ± 0%       ~     (all equal)
Medium/goleveldb-100000-100-16-40/iteration-fast-16                       523k ± 0%      523k ± 0%       ~     (p=0.444 n=5+5)
Medium/goleveldb-100000-100-16-40/iteration-slow-16                      4.71M ± 0%     4.71M ± 0%       ~     (p=0.200 n=4+4)
Medium/goleveldb-100000-100-16-40/update-16                                562 ±14%       566 ±16%       ~     (p=0.690 n=5+5)
Medium/goleveldb-100000-100-16-40/block-16                               72.9k ± 3%     75.0k ± 4%       ~     (p=0.151 n=5+5)

Clearly, having cache in nodedb does make a big difference

@faddat
Copy link
Member

faddat commented Mar 4, 2022

science! that is why it's called a lab <3

Btw, nice benchmarks sir

@p0mvn
Copy link
Member Author

p0mvn commented Mar 4, 2022 •
edited
Loading

It is surprising to me that the decrease in performance affected only query-hits-fast-16, not iterations or "not in tree queries". query-hits-fast-16 are the queries that guarantee for the keys we are querying for to be present in the tree. The reduced performance with no cache implies that there is no substitute cache in goleveldb.

Our initial guess was that the cache in nodedb might be redundant since, from my understanding, there are 2 "in-memory" features in goleveldb:

  1. first, we write to memdb, and when it fills up we flush it to disk
    • I was surprised to see that Get() did not go to membdb. On the other hand, it went to disk directly
  2. There should be an 8 MB default LRU cache in goleveldb.

I did the pprof of queries-hits-fast-16. The relevant path is the rightmost in light red:
image

According to pprof, Get() does not indeed go to memdb and, instead, directly to disk. This can be seen in this call to get() where the provided as a parameter auxm *memdb.DB is nil. It is not nil only when transactions are used.

From pprof, we also see that there are calls to Cache.get() at the bottom of the graph. I investigated that call and found out that there is a configuration option DontFillCache. It seems to be false by default (what we need).

At this point, it is unclear to me why I can't observe cache being utilized in golebeldb.

I am going to pause the work on this issue since the scope has changed. I will come back to it when I'm working on osmosis-labs/iavl#32 which is about investigating goleveldb configs. Once I understand some goleveldb configs, I will see if it makes removing fast node cache from nodedb.

My branch with no fast cache in node db: roman/remove-fast-cache

The goleveldb settings that get enabled by default are:

{
   "AltFilters":"nil",
   "BlockCacher":"nil",
   "BlockCacheCapacity":0,
   "BlockCacheEvictRemoved":false,
   "BlockRestartInterval":0,
   "BlockSize":0,
   "CompactionExpandLimitFactor":0,
   "CompactionGPOverlapsFactor":0,
   "CompactionL0Trigger":0,
   "CompactionSourceLimitFactor":0,
   "CompactionTableSize":0,
   "CompactionTableSizeMultiplier":0,
   "CompactionTableSizeMultiplierPerLevel":"nil",
   "CompactionTotalSize":0,
   "CompactionTotalSizeMultiplier":0,
   "CompactionTotalSizeMultiplierPerLevel":"nil",
   "Comparer":"nil",
   "Compression":0,
   "DisableBufferPool":false,
   "DisableBlockCache":false,
   "DisableCompactionBackoff":false,
   "DisableLargeBatchTransaction":false,
   "DisableSeeksCompaction":false,
   "ErrorIfExist":false,
   "ErrorIfMissing":false,
   "Filter":"nil",
   "IteratorSamplingRate":0,
   "NoSync":false,
   "NoWriteMerge":false,
   "OpenFilesCacher":"nil",
   "OpenFilesCacheCapacity":0,
   "ReadOnly":false,
   "Strict":58,
   "WriteBuffer":0,
   "WriteL0PauseTrigger":0,
   "WriteL0SlowdownTrigger":0,
   "FilterBaseLg":0
}

I can see in the defaults that the BlockCacher is nil. However, I think that this block cache is different from the in-memory cache. Need to investigate more

@p0mvn
Copy link
Member Author

p0mvn commented Mar 4, 2022

I created a manual test where I set only 2 keys i.e.:

d.Set([]byte("key1"), []byte("value1"))
d.Set([]byte("key2"), []byte("value2"))

Then, to simulate the query-hits-fast-16 bench I attempted to retrieve "key1":

_, err = d.Get([]byte("key1"))
require.NoError(t, err)

Contrary to the pprof results from the benchmark, it went to the in-memory db within goleveldb .
We might need to configure this in-memory db to work with more data (or key-value pairs)

@p0mvn
Copy link
Member Author

p0mvn commented Mar 28, 2022

This issue is resolved, reconfigured fast cache to a smaller value

@p0mvn p0mvn closed this as completed Mar 28, 2022
@p0mvn p0mvn mentioned this issue Jun 16, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@p0mvn p0mvn

Labels
feature New feature or request needs-investigation T:task ⚙️ A task belongs to a story
Projects
Osmosis Chain Development
Archived in project
Milestone
No milestone
Development

No branches or pull requests
2 participants
@faddat @p0mvn