Supporting multiple online feature stores

Is your feature request related to a problem? Please describe.
We are seeing that ML feature teams want to own their feature projections and not rely on a single global online feature store with the most common reason being reduction of blast radius.

Describe the solution you'd like
A way to create projections of a subset of feature views to specific online stores.

Describe alternatives you've considered
I was pointed towards attempting to implement the requested feature using a custom provider which I managed to get working, albeit rather hacky. I've only focused on fanning out the writing of features.

class ManyOnlineStoresPassthroughConfig(RepoConfig):
    online_stores: List[dict]

class ManyOnlineStoresPassthroughProvider(PassthroughProvider):
    def online_write_batch(self, config: ManyOnlineStoresPassthroughConfig, table: FeatureView, *args, **kwargs) -> None:
        """
        Writes feature view to all subscribers
        """
        feature_view_subscribers = [store for store in config.online_stores if table.name in store["view_subscriptions"]]

        for store in feature_view_subscribers:
            print(f"{table.name} online write to subscriber {store['name']}")

            config.online_store = get_online_config_from_type(store["type"])(**store)
            self.online_store = get_online_store_from_config(config.online_store)

            return super().online_write_batch(config, table, *args, **kwargs)

With the config update of

project: aaaa
registry: gs://temp-feast-dev/dev/registry.db
provider: providers.custom_provider.ManyOnlineStoresPassthroughProvider

offline_store:
  type: bigquery
  dataset: temp_feast-dev
  project_id: abc
  location: EU

online_stores:
  - name: "lovely"
    type: redis
    redis_type: redis
    connection_string: "blah"
    view_subscriptions: ["feature_view_1"]

  - name: "lovely"
    type: redis
    redis_type: redis
    connection_string: "abc"
    view_subscriptions: ["feature_view_1", "feature_view_2"]

There's still some unknowns around retrieving the online features as we'd need a way of knowing what projection should be used.

Happy to help wherever I can!

@alexjajones

Thanks for this @alexjajones. Have you had a look at what we do in Tecton? We basically have the online store defined within the feature repo. In Feast this would look like

feature_view.yaml

# I would use a dict to avoid name collissions
online_stores:
  team_1_store:
    type: redis
    redis_type: redis
    connection_string: "blah"

  team_2_store:
    type: redis
    redis_type: redis
    connection_string: "abc"

repo.py

fv_online_store_1 = FeatureView( ..., online_store="team_1_store")
fv_online_store_2 = FeatureView( ..., online_store="team_2_store")

Or have you found that the same feature views are subscribed to by multiple teams?

Something like the following?

class ManyOnlineStoresPassthroughConfig(RepoConfig):
    online_stores: Dict

class ManyOnlineStoresFeatureView(FeatureView):
    online_stores: List[str]

class SpecificPassthroughProvider(PassthroughProvider):

    def online_write_batch(self, config: ManyOnlineStoresPassthroughConfig, table: ManyOnlineStoresFeatureView, *args, **kwargs) -> None:
        for store_name, store_details in config.online_stores.items():

            if store_name in table.online_stores:

                config.online_store = get_online_config_from_type(store_details["type"])(**store_details)
                self.online_store = get_online_store_from_config(config.online_store)

                return super().online_write_batch(config, table, *args, **kwargs)

fv_online_store_1 = FeatureView( ..., online_stores=["team_1_store"])
fv_online_store_2 = FeatureView( ..., online_stores=["team_2_store"])

online_stores:
  team_1_store:
    type: redis
    redis_type: redis
    connection_string: "blah"

  team_2_store:
    type: redis
    redis_type: redis
    connection_string: "abc"`

Or have you found that the same feature views are subscribed to by multiple teams?

I could see a case where many teams would want to subscribe to a core customers feature view and then subscribe to other more specific views for the use case.

Saying that, this approach could lead to a bunch of features in the online store which aren't used ... so maybe it would be worth creating a custom feature view that pulls in only the required core features.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

This was implemented via Hybrid Online Stores #5423

Navigation Menu

Repository navigation

Supporting multiple online feature stores #2383

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Supporting multiple online feature stores #2383

Description

Activity

woop commented on Mar 7, 2022

alexjajones commented on Mar 9, 2022

stale commented on Dec 20, 2022

franciscojavierarceo commented on Dec 16, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions