Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(rocksdb): Adapt prefix bloom filter to speedup scans by hashkey #438

Merged
merged 5 commits into from
Dec 19, 2019
Merged

feat(rocksdb): Adapt prefix bloom filter to speedup scans by hashkey #438

merged 5 commits into from
Dec 19, 2019
+228 −20

Conversation

acelyc111
Copy link
Member

@acelyc111 acelyc111 commented Dec 7, 2019 •
edited
Loading

What problem does this PR solve?

Improve the performance of scans by hashkeys
Related issue: #421

测试Case 读写比 运行时长 读QPS 读Avg延迟 读P99延迟 写TPS 写Avg延迟 写P99延迟
1.12版本                
(1)数据加载: 3客户端*10线程 0:1 2.05 - - - 40439 739 2995
(2)​读写同时: 3客户端*15线程 ​1:3 1.3 16022 309 759 48078 830 3995
​(3)读写同时: 3客户端*30线程 ​30:1 0.33 244392 346 652 8137 731 2995
(4)数据只读: 6客户端*100线程 1:0 0.25 672737 914 3205 - - -
(5)数据只读: 12客户端*100线程                
(6)数据加载: 6客户端10线程multi 0:1 1.57 - - - 55871 1122 5155
(7)数据只读: 6客户端10线程range 1:0 0.68 245237 245 464 - - -
(8)数据只读: 6客户端10线程range*10%命中率 1:0 0.67 252533 240 551 - - -
(9)数据只读: 6客户端10线程range*1%命中率 1:0 0.58 290780 206 383 - - -
                 
prefix 版本                
(1)数据加载: 3客户端*10线程 0:1 1.99 -   - 41876 714 2985
(2)​读写同时: 3客户端*15线程 ​1:3 1.28 16276 303 818 48828 816 4061
​(3)读写同时: 3客户端*30线程 ​30:1 0.31 260147(+6.4%) 322 664 8670 715 3227
(4)数据只读: 6客户端*100线程 1:0 0.21 673418 770 2445 - - -
(5)数据只读: 12客户端*100线程                
(6)数据加载: 6客户端10线程multi 0:1 1.5 - - -   1072 5099
(7)数据只读: 6客户端10线程range 1:0 0.66 263446(+7.4%) 229 394 - - -
(8)数据只读: 6客户端10线程range*10%命中率 1:0 0.57 295226(+16.9%) 201 322 - - -
(9)数据只读: 6客户端10线程range*1%命中率 1:0 0.55 318697(+9.6%) 188 322 - - -

What is changed and how it works?

Add config:

[pegasus.server]
  rocksdb_filter_type = prefix

Check List

Tests

  • Unit test
    Yes
  • Integration test
    Yes
  • Manual test (add detailed scripts or steps below)
    Yes
    Steps:
  1. Set rocksdb_filter_type = common
  2. Write data set A
  3. Check data set A
  4. Set rocksdb_filter_type = prefix and restart replica server
  5. Write data set B
  6. Check data set A and B
  7. Set rocksdb_filter_type = common and restart replica server
  8. Check data set A and B

Code changes

  • Has persistent data change
    Yes, but it's compatible.

Related changes

  • Need to update the documentation
    Yes
  • Need to be included in the release note
    Yes

@acelyc111 acelyc111 changed the title (feat)rocksdb: Adapt prefix bloom filter to speedup scan by hashkey feat(rocksdb): Adapt prefix bloom filter to speedup scan by hashkey Dec 7, 2019
@weekly-digest weekly-digest bot mentioned this pull request Dec 8, 2019
Closed
@acelyc111 acelyc111 changed the title feat(rocksdb): Adapt prefix bloom filter to speedup scan by hashkey feat(rocksdb): Adapt prefix bloom filter to speedup scans by hashkey Dec 9, 2019
neverchanje
neverchanje reviewed Dec 9, 2019
View reviewed changes
src/server/config.min.ini Outdated Show resolved Hide resolved
run.sh Show resolved Hide resolved
src/server/hashkey_transform.h Show resolved Hide resolved
@acelyc111 acelyc111 closed this Dec 10, 2019
@acelyc111 acelyc111 reopened this Dec 10, 2019
@acelyc111 acelyc111 added submodule/rocksdb type/performance performance optimization or tunning labels Dec 12, 2019
@weekly-digest weekly-digest bot mentioned this pull request Dec 15, 2019
Closed
hycdong
hycdong reviewed Dec 17, 2019
View reviewed changes
src/server/config.ini Outdated Show resolved Hide resolved
hycdong
hycdong previously approved these changes Dec 17, 2019
View reviewed changes
neverchanje
neverchanje reviewed Dec 18, 2019
View reviewed changes
src/server/pegasus_server_impl.cpp Outdated Show resolved Hide resolved
src/server/pegasus_server_impl.cpp Outdated Show resolved Hide resolved
run.sh Show resolved Hide resolved
@neverchanje neverchanje added the type/config-change Added or modified configuration that should be noted on release note of new version. label Dec 18, 2019
neverchanje
neverchanje reviewed Dec 18, 2019
View reviewed changes
src/server/test/hashkey_transform_test.cpp Outdated Show resolved Hide resolved
src/server/pegasus_server_impl.cpp Outdated Show resolved Hide resolved
src/server/pegasus_server_impl.cpp Outdated Show resolved Hide resolved
@acelyc111 acelyc111 merged commit cecef44 into apache:master Dec 19, 2019
@weekly-digest weekly-digest bot mentioned this pull request Dec 22, 2019
Closed
acelyc111 added a commit that referenced this pull request Dec 31, 2019
@acelyc111
feat(rocksdb): Adapt prefix bloom filter to speedup scans by hashkey (#…
12481a8 
…438)
@neverchanje
Copy link
Contributor

neverchanje commented Feb 27, 2020 •
edited
Loading

测试Case 读写比 运行时长 读QPS 读Avg延迟 读P99延迟 写TPS 写Avg延迟 写P99延迟
1.12版本                
(7)数据只读: 6客户端10线程range 1:0 0.68 245237 245 464 - - -
(8)数据只读: 6客户端10线程range*10%命中率 1:0 0.67 252533 240 551 - - -
(9)数据只读: 6客户端10线程range*1%命中率 1:0 0.58 290780 206 383 - - -
prefix版本                
(7)数据只读: 6客户端10线程range 1:0 0.66 263446(+7.4%) 229 394 - - -
(8)数据只读: 6客户端10线程range*10%命中率 1:0 0.57 295226(+16.9%) 201 322 - - -
(9)数据只读: 6客户端10线程range*1%命中率 1:0 0.55 318697(+9.6%) 188 322 - - -

acelyc111 pushed a commit that referenced this pull request Jun 23, 2022
build(deps): use cmake ExternalProject to manage dependencies (#438)
aef76b1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hycdong hycdong hycdong left review comments

@foreverneverer foreverneverer foreverneverer approved these changes

@neverchanje neverchanje neverchanje approved these changes

Assignees
No one assigned
Labels
1.12.2 type/config-change Added or modified configuration that should be noted on release note of new version. type/performance performance optimization or tunning
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@acelyc111 @neverchanje @hycdong @foreverneverer