Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Java Feature Server] Use hgetall in redis connector when number of retrieved fields is big enough #2159

Merged
merged 8 commits into from
Dec 20, 2021

Conversation

pyalex
Copy link
Collaborator

@pyalex pyalex commented Dec 17, 2021

Signed-off-by: pyalex moskalenko.alexey@gmail.com

What this PR does / why we need it:

According to my analysis after the number of fields passed to hmget surpasses some value (roughly 50) - hgetall becomes a much more efficient way to retrieve the same data. With 250 fields - hgetall is 5x faster then hmget.

Which issue(s) this PR fixes:

Fixes #

Does this PR introduce a user-facing change?:

none

Signed-off-by: pyalex <moskalenko.alexey@gmail.com>
Signed-off-by: pyalex <moskalenko.alexey@gmail.com>
Signed-off-by: pyalex <moskalenko.alexey@gmail.com>
Signed-off-by: pyalex <moskalenko.alexey@gmail.com>
Signed-off-by: pyalex <moskalenko.alexey@gmail.com>
@codecov-commenter
Copy link

codecov-commenter commented Dec 17, 2021

Codecov Report

Merging #2159 (b3edfcf) into master (a753e73) will decrease coverage by 25.65%.
The diff coverage is n/a.

Impacted file tree graph

@@             Coverage Diff             @@
##           master    #2159       +/-   ##
===========================================
- Coverage   84.40%   58.75%   -25.66%     
===========================================
  Files         101      101               
  Lines        8098     8107        +9     
===========================================
- Hits         6835     4763     -2072     
- Misses       1263     3344     +2081     
Flag Coverage Δ
integrationtests ?
unittests 58.75% <ø> (-0.06%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
.../integration/online_store/test_universal_online.py 15.34% <0.00%> (-82.80%) ⬇️
.../integration/online_store/test_online_retrieval.py 17.39% <0.00%> (-82.61%) ⬇️
...fline_store/test_universal_historical_retrieval.py 19.18% <0.00%> (-80.23%) ⬇️
sdk/python/tests/utils/online_read_write_test.py 20.68% <0.00%> (-79.32%) ⬇️
...gration/registration/test_feature_service_apply.py 31.25% <0.00%> (-68.75%) ⬇️
sdk/python/feast/infra/online_stores/redis.py 26.97% <0.00%> (-67.11%) ⬇️
sdk/python/tests/data/data_creator.py 34.78% <0.00%> (-65.22%) ⬇️
sdk/python/tests/integration/e2e/test_usage_e2e.py 35.00% <0.00%> (-65.00%) ⬇️
...s/integration/registration/test_universal_types.py 36.12% <0.00%> (-62.59%) ⬇️
...istration/test_universal_odfv_feature_inference.py 38.29% <0.00%> (-61.71%) ⬇️
... and 49 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a753e73...b3edfcf. Read the comment docs.

Signed-off-by: pyalex <moskalenko.alexey@gmail.com>
Signed-off-by: pyalex <moskalenko.alexey@gmail.com>
@pyalex pyalex changed the title [WIP][Java Feature Server] Use hgetall in redis connector when number of retrieved fields is big enough [Java Feature Server] Use hgetall in redis connector when number of retrieved fields is big enough Dec 17, 2021

// Number of fields that controls whether to use hmget or hgetall was discovered empirically
// Could be potentially tuned further
if (retrieveFields.size() < 50) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: maybe make this a constant at the beginning of the file so it's better visible?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added constant

Copy link
Collaborator

@tsotnet tsotnet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me overall

@feast-ci-bot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: pyalex, tsotnet

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Signed-off-by: pyalex <moskalenko.alexey@gmail.com>
@pyalex pyalex merged commit 033659e into feast-dev:master Dec 20, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants