Skip to content

[Java Feature Server] Use hgetall in redis connector when number of retrieved fields is big enough - #2159

Merged
pyalex merged 8 commits into
feast-dev:masterfrom
pyalex:java-hgetall
Dec 20, 2021
Merged

[Java Feature Server] Use hgetall in redis connector when number of retrieved fields is big enough#2159
pyalex merged 8 commits into
feast-dev:masterfrom
pyalex:java-hgetall

Conversation

@pyalex

@pyalex pyalex commented Dec 17, 2021

Copy link
Copy Markdown
Collaborator

Signed-off-by: pyalex moskalenko.alexey@gmail.com

What this PR does / why we need it:

According to my analysis after the number of fields passed to hmget surpasses some value (roughly 50) - hgetall becomes a much more efficient way to retrieve the same data. With 250 fields - hgetall is 5x faster then hmget.

Which issue(s) this PR fixes:

Fixes #

Does this PR introduce a user-facing change?:

none

Signed-off-by: pyalex <moskalenko.alexey@gmail.com>
Signed-off-by: pyalex <moskalenko.alexey@gmail.com>
Signed-off-by: pyalex <moskalenko.alexey@gmail.com>
Signed-off-by: pyalex <moskalenko.alexey@gmail.com>
Signed-off-by: pyalex <moskalenko.alexey@gmail.com>
@codecov-commenter

codecov-commenter commented Dec 17, 2021

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.39%. Comparing base (a753e73) to head (6d51eaf).
⚠️ Report is 2386 commits behind head on master.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2159      +/-   ##
==========================================
- Coverage   84.40%   84.39%   -0.01%     
==========================================
  Files         101      101              
  Lines        8098     8133      +35     
==========================================
+ Hits         6835     6864      +29     
- Misses       1263     1269       +6     
Flag Coverage Δ
integrationtests 74.39% <ø> (-0.27%) ⬇️
unittests 58.73% <ø> (-0.07%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Signed-off-by: pyalex <moskalenko.alexey@gmail.com>
Signed-off-by: pyalex <moskalenko.alexey@gmail.com>
@pyalex pyalex changed the title [WIP][Java Feature Server] Use hgetall in redis connector when number of retrieved fields is big enough [Java Feature Server] Use hgetall in redis connector when number of retrieved fields is big enough Dec 17, 2021

// Number of fields that controls whether to use hmget or hgetall was discovered empirically
// Could be potentially tuned further
if (retrieveFields.size() < 50) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: maybe make this a constant at the beginning of the file so it's better visible?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added constant

@tsotnet tsotnet left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me overall

@feast-ci-bot

Copy link
Copy Markdown
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: pyalex, tsotnet

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Signed-off-by: pyalex <moskalenko.alexey@gmail.com>
@pyalex
pyalex merged commit 033659e into feast-dev:master Dec 20, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants