Add support of using hyper clock cache as the block cache #2031

JxLi0921 · 2024-01-20T02:39:26Z

This is related to this issue #2022 . I will soon post my simple benchmark result for this PR below.

JxLi0921 · 2024-01-20T02:44:09Z

The script for pressing data into kvrocks:

memtier_benchmark -t 4 -c 20 -h 127.0.0.1 -p 6666 --distinct-client-seed --key-minimum=1000000000 --key-maximum=10000000000 --random-data --key-prefix="" --data-size=50 -n 2000000 --pipeline=100 --command "mset key data key data key data key data key data key data key data key data key data key data"

The script for benchmarking: where the READ_RATIO is 0, 50, 90 or 100, respectively.

memtier_benchmark -t 4 -c 20 -s 127.0.0.1 -p 6666 --distinct-client-seed --key-minimum=1000000000 --key-maximum=10000000000 --key-prefix="" -n 10000 --command "get key" --command-ratio=READ_RATIO --command "set key value" --command-ratio=100-READ_RATIO

Here is the P99, P999 and P9999 latency with difference read ratios for the SET and GET command in the benchmark above.

SET:

GET:

wanghenshui · 2024-01-20T07:20:10Z

--data-size=50 -n 2 000 000

50 * 2,000,000 * 20(c) = 2,000,000,000

how much rocksdb sst data your make, 2G? please make at least 10G data

mapleFU · 2024-01-20T07:45:15Z

I think the clock might has better concurrency and perhaps a bit worse cache hit rate. We can also report the cache hit rate etc for analyzing the performance here

mapleFU

+1 for this pr.
And maybe we can dive into benchmark result and find some reason of performance🤔

src/storage/storage.cc

JxLi0921 · 2024-01-20T08:12:14Z

--data-size=50 -n 2 000 000
50 * 2,000,000 * 20(c) = 2,000,000,000
how much rocksdb sst data your make, 2G? please make at least 10G data

@wanghenshui Thanks for your review! The command I used to press the data into kvrocks is mset key data * 10, so maybe it's 20G instead of 2G? I am not convinced with my understanding, please correct me if I am wrong.

JxLi0921 · 2024-01-20T08:23:49Z

I think the clock might has better concurrency and perhaps a bit worse cache hit rate. We can also report the cache hit rate etc for analyzing the performance here

Thanks for your review! I would add these statistics later today.

mapleFU · 2024-01-20T08:25:53Z

As for the statistics, you can refer Server::GetRocksDBInfo for help

PragmaTwice · 2024-01-20T10:12:21Z

kvrocks.conf

+# Specify the type of cache used in the block cache.
+# Accept value: "lru", "hcc"
+# default lru
+rocksdb.block_cache_type lru


Could you explain the meaning of lru and hcc for users in the comment>

PragmaTwice · 2024-01-20T10:16:13Z

src/storage/storage.cc

-    options.row_cache = rocksdb::NewLRUCache(config_->rocks_db.row_cache_size * MiB);
+    // Parameters for the rocksdb::LRUCacheOptions here are set in line with the default parameters for the
+    // rocksdb::NewLRUCache function, which is now deprecated
+    rocksdb::LRUCacheOptions lru_cache_options(config_->rocks_db.row_cache_size * MiB, -1, false, 0.5);


Could we add some named constants for these default arguments (e.g. -1, false, 0.5)?

I have did this, but I am not sure whether the names of my constants are proper.

PragmaTwice · 2024-01-20T10:16:38Z

src/storage/storage.cc

+    rocksdb::LRUCacheOptions lru_cache_options(block_cache_size, -1, false, 0.75);
+    shared_block_cache = lru_cache_options.MakeSharedCache();
+  } else {
+    rocksdb::HyperClockCacheOptions hcc_cache_options(block_cache_size, 0);


ditto for 0 here.

git-hulk · 2024-01-20T15:42:49Z

@JxLi0921 Great, thanks for your PR. I'm not sure if the QPS throughput is close between Hyper Clock Cache and LRU since I didn't see this in previous report.

JxLi0921 · 2024-01-24T07:43:34Z

Sorry for late update. I found that the way I use memtier_benchmark is totally wrong before.

Here is my current script for putting data and running benchmark:

Putting data into kvrocks:

memtier_benchmark -t 8 -c 20 -h 127.0.0.1 -p 6666 --distinct-client-seed --key-minimum=1000000000 --key-maximum=10000000000 --random-data --key-prefix="" --data-size
=50 -n 2000000 --pipeline=100 --command "mset __key__ __data__ __key__ __data__ __key__ __data__  __key__ __data__ __key__ __data__ __key__ __data__ __key__ __data__ __key__ 
__data__ __key__ __data__ __key__ __data__"

I failed to run the script above to complete because I am almost run out of disk space. After running the script above for about 20% of progress, the keyspace statistics of kvrocks is as follows.

#  Keyspace
# Last DBSIZE SCAN time: Wed Jan 24 12:04:59 2024
db0:keys=341377306,expires=0,avg_ttl=0,expired=0
sequence:615748420
used_db_size:18388474802
max_db_size:0
used_percent: 0%

Running benchmark:

memtier_benchmark -t 4 -c 20 -s 127.0.0.1 -p 6666 --distinct-client-seed --key-minimum=1000000000  --key-maximum=10000000000 --key-prefix="" -n 50000 --ratio={100-read_ratio}:{read_ratio}

where the read_ratio above is one of 0, 50, 90 and 100.

Result:

GET Latency:

SET Latency:

GET Ops/sec

SET Ops/sec

Overall(Get+Set) Ops/sec

According to the results listed above, it seems that HCC does outperform LRU for latency and throughput in this simple benchmark.

Block Cache Hit Ratio:

I'm not sure whether it's normal to have the block cache hit ratios to be almost 100. Should I use a smaller block cache size? I used 4096MB block cache size in this benchmark, which is kvrocks' default configuration.

mapleFU

Thanks for the investigate. I may go for this this weekend. At lease we can first checkin this patch?

Also @JxLi0921 would you mind tell the CPU and disk you're using?

git-hulk

Cool, LGTM

JxLi0921 · 2024-01-24T08:55:47Z

would you mind tell the CPU and disk you're using?

@mapleFU

CPU: AMD Ryzen 6800H
Disk: SAMSUNG MZVL2512HCJQ-00B00

JxLi0921 added 3 commits January 20, 2024 10:31

allow using hyper clock cache

57e07ca

remove redundant include

5aa2a89

fix: hcc cache cannot be used in row cache

5c12185

JxLi0921 added 2 commits January 20, 2024 10:48

fix: remove redundant include

20ddbf7

fix: remove one more redundant include

717cebe

mapleFU reviewed Jan 20, 2024

View reviewed changes

src/storage/storage.cc Outdated Show resolved Hide resolved

NewLRUCache -> LRUCacheOptions

5c76a99

fix CI build failed

a39e51b

PragmaTwice reviewed Jan 20, 2024

View reviewed changes

JxLi0921 added 4 commits January 20, 2024 19:31

make it build with speedb

ef2c15f

add more comments to the configuration file

f7dce67

add named constants in creating cache

9267b3d

row cache

07b3681

mapleFU and others added 4 commits January 22, 2024 14:57

Merge branch 'unstable' into use_hyper_clock_cache

ccd70fc

Merge branch 'unstable' into use_hyper_clock_cache

7bd969f

Merge branch 'unstable' into use_hyper_clock_cache

d23ef48

Merge branch 'unstable' into use_hyper_clock_cache

8f47e4c

mapleFU approved these changes Jan 24, 2024

View reviewed changes

git-hulk approved these changes Jan 24, 2024

View reviewed changes

mapleFU merged commit 02a6240 into apache:unstable Jan 24, 2024
29 checks passed

git-hulk mentioned this pull request Mar 9, 2024

Allow to use hyper clock cache to replace the LRU cache #2022

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support of using hyper clock cache as the block cache #2031

Add support of using hyper clock cache as the block cache #2031

JxLi0921 commented Jan 20, 2024

JxLi0921 commented Jan 20, 2024

wanghenshui commented Jan 20, 2024

mapleFU commented Jan 20, 2024

mapleFU left a comment

JxLi0921 commented Jan 20, 2024

JxLi0921 commented Jan 20, 2024

mapleFU commented Jan 20, 2024

PragmaTwice Jan 20, 2024

PragmaTwice Jan 20, 2024

JxLi0921 Jan 20, 2024 •

edited

Loading

PragmaTwice Jan 20, 2024

git-hulk commented Jan 20, 2024 •

edited

Loading

JxLi0921 commented Jan 24, 2024 •

edited

Loading

mapleFU left a comment

git-hulk left a comment

JxLi0921 commented Jan 24, 2024 •

edited

Loading

Add support of using hyper clock cache as the block cache #2031

Add support of using hyper clock cache as the block cache #2031

Conversation

JxLi0921 commented Jan 20, 2024

JxLi0921 commented Jan 20, 2024

wanghenshui commented Jan 20, 2024

mapleFU commented Jan 20, 2024

mapleFU left a comment

Choose a reason for hiding this comment

JxLi0921 commented Jan 20, 2024

JxLi0921 commented Jan 20, 2024

mapleFU commented Jan 20, 2024

PragmaTwice Jan 20, 2024

Choose a reason for hiding this comment

PragmaTwice Jan 20, 2024

Choose a reason for hiding this comment

JxLi0921 Jan 20, 2024 • edited Loading

Choose a reason for hiding this comment

PragmaTwice Jan 20, 2024

Choose a reason for hiding this comment

git-hulk commented Jan 20, 2024 • edited Loading

JxLi0921 commented Jan 24, 2024 • edited Loading

mapleFU left a comment

Choose a reason for hiding this comment

git-hulk left a comment

Choose a reason for hiding this comment

JxLi0921 commented Jan 24, 2024 • edited Loading

JxLi0921 Jan 20, 2024 •

edited

Loading

git-hulk commented Jan 20, 2024 •

edited

Loading

JxLi0921 commented Jan 24, 2024 •

edited

Loading

JxLi0921 commented Jan 24, 2024 •

edited

Loading