Skip to content

Add support for Redis and Redis Cluster #1511

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
feast-ci-bot merged 22 commits into from
Jun 9, 2021
Merged

Add support for Redis and Redis Cluster #1511

Changes from 1 commit
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
d97aaea
Add support for Redis as online store
qooba Apr 25, 2021
ae9ed8e
Add support for Redis as online store
qooba Apr 25, 2021
e165f00
Add redis provider
qooba Apr 26, 2021
df8f3cb
Add redis online provider
qooba Apr 27, 2021
6bea80d
Add redis online provider
qooba Apr 27, 2021
5745ff7
Add redis online provider
qooba Apr 27, 2021
a3c4f3b
Add redis online provider
qooba Apr 28, 2021
e434e59
Add redis online provider - integration tests
qooba May 6, 2021
8ccfc6b
Add redis online provider - integration tests
qooba May 6, 2021
be9a17f
Add redis provider
qooba May 12, 2021
7606fe7
Add redis provider
qooba May 12, 2021
2a95e3e
correct redis provider
qooba May 24, 2021
435c856
correct redis provider
qooba May 26, 2021
d329439
correct redis provider test
qooba May 26, 2021
b5269e7
correct redis provider
qooba May 30, 2021
f602b9a
correct redis provider
qooba May 30, 2021
bbde0b0
change redis connection string
qooba Jun 2, 2021
d86f23c
change redis connection string
qooba Jun 2, 2021
b0c8101
fix redis materialize
qooba Jun 7, 2021
52232cc
Rename connection string and set defaults
woop Jun 8, 2021
53f8f72
Fix argument in redis test
woop Jun 9, 2021
e12ecff
Fix broken connection string
woop Jun 9, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add redis provider 
Signed-off-by: qooba <dev@qooba.net>
  • Loading branch information
@qooba @woop
qooba authored and woop committed Jun 9, 2021
commit be9a17fd822827c85d026c2f2d08a624ef32b557
9 changes: 8 additions & 1 deletion sdk/python/feast/infra/redis_provider.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
from google.protobuf.timestamp_pb2 import Timestamp
from redis import Redis
from rediscluster import RedisCluster
from tqdm import tqdm

from feast import FeatureTable, utils
from feast.entity import Entity
Expand Down Expand Up @@ -96,6 +97,8 @@ def online_write_batch(
entity_hset[f_key] = val.SerializeToString()

client.hset(redis_key_bin, mapping=entity_hset)
if progress:
progress(1)

def online_read(
self,
Expand Down Expand Up @@ -148,6 +151,7 @@ def materialize_single_feature_view(
end_date: datetime,
registry: Registry,
project: str,
tqdm_builder: Callable[[int], tqdm],
) -> None:
entities = []
for entity_name in feature_view.entities:
Expand Down Expand Up @@ -180,7 +184,10 @@ def materialize_single_feature_view(
join_keys = [entity.join_key for entity in entities]
rows_to_write = _convert_arrow_to_proto(table, feature_view, join_keys)

self.online_write_batch(project, feature_view, rows_to_write, None)
with tqdm_builder(len(rows_to_write)) as pbar:
self.online_write_batch(
project, feature_view, rows_to_write, lambda x: pbar.update(x)
)

feature_view.materialization_intervals.append((start_date, end_date))
registry.apply_feature_view(feature_view, project)
Expand Down