Skip to content

Commit 06aa2e5

Browse files
Merge branch 'master' into feat/operator-jwks-tunables
2 parents ecd0c59 + 4efb86c commit 06aa2e5

9 files changed

Lines changed: 336 additions & 19 deletions

File tree

docs/SUMMARY.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -149,6 +149,7 @@
149149
* [Snowflake](reference/online-stores/snowflake.md)
150150
* [Redis](reference/online-stores/redis.md)
151151
* [Dragonfly](reference/online-stores/dragonfly.md)
152+
* [Valkey](reference/online-stores/valkey.md)
152153
* [Datastore](reference/online-stores/datastore.md)
153154
* [DynamoDB](reference/online-stores/dynamodb.md)
154155
* [Bigtable](reference/online-stores/bigtable.md)

docs/getting-started/components/authz_manager.md

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -122,6 +122,19 @@ A token whose `aud` (or `iss`) claim does not match is rejected at authenticatio
122122
Set these to the values your IdP puts **in the token itself**, which are not always the ones in the discovery document. For example, Microsoft Entra ID commonly issues v1.0 tokens (`iss: https://sts.windows.net/<tenant-id>/`, `aud: api://<app-id-uri>`) even when `auth_discovery_url` points at the v2.0 endpoint. That setup keeps working with these options unset, or set to the v1.0 values — but copying the v2.0 issuer from the discovery document would reject every v1.0 token.
123123
{% endhint %}
124124

125+
To validate token signatures the server fetches the provider's JWKS document and caches it, refetching when the cache expires or when a token presents an unknown key id. Two options tune that behavior:
126+
127+
```yaml
128+
auth:
129+
type: oidc
130+
client_id: _CLIENT_ID_
131+
auth_discovery_url: https://login.example.com/.well-known/openid-configuration
132+
jwks_cache_lifespan_seconds: 300 # default; how long the fetched key set is reused
133+
jwks_request_timeout_seconds: 10 # default; network timeout for the JWKS fetch
134+
```
135+
136+
`jwks_cache_lifespan_seconds` also bounds how long a key the provider has **revoked** continues to validate tokens, so lower it if your provider rotates or revokes aggressively; each reduction costs proportionally more JWKS fetches. Key rotations that introduce a new key id are picked up immediately regardless of this setting, because an unknown key id triggers a refetch. `jwks_request_timeout_seconds` bounds how long an unresponsive provider can block request serving. Both must be greater than zero.
137+
125138
#### Client-Side Configuration
126139

127140
The client supports multiple token source modes. The SDK resolves tokens in the following priority order:

docs/reference/online-stores/README.md

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,10 @@ Please see [Online Store](../../getting-started/components/online-store.md) for
2222
[dragonfly.md](dragonfly.md)
2323
{% endcontent-ref %}
2424

25+
{% content-ref url="valkey.md" %}
26+
[valkey.md](valkey.md)
27+
{% endcontent-ref %}
28+
2529
{% content-ref url="datastore.md" %}
2630
[datastore.md](datastore.md)
2731
{% endcontent-ref %}
@@ -31,7 +35,7 @@ Please see [Online Store](../../getting-started/components/online-store.md) for
3135
{% endcontent-ref %}
3236

3337
{% content-ref url="bigtable.md" %}
34-
[bigtable.md](mysql.md)
38+
[bigtable.md](bigtable.md)
3539
{% endcontent-ref %}
3640

3741
{% content-ref url="postgres.md" %}
Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
# Valkey online store
2+
3+
## Description
4+
5+
[Valkey](https://valkey.io/) is an open source (BSD-3-Clause), high-performance key/value datastore hosted by the Linux Foundation, created as a community fork of Redis. It maintains compatibility with the Redis wire protocol, so it can act as a drop-in replacement for Redis. Valkey is also offered as a managed engine by major cloud providers (for example, Amazon ElastiCache for Valkey).
6+
7+
Similar to Redis and [Dragonfly](dragonfly.md), Valkey can be used as an online feature store for Feast: Feast's Redis online store only issues core commands (hash reads/writes, scans, key expiry, pipelines), all of which Valkey implements.
8+
9+
Feast's standard online store operations have been verified against Valkey 8.1: `feast apply`, `feast materialize`, online retrieval via `get_online_features`, `feast teardown`, and key expiry via the `key_ttl_seconds` option. Features that depend on Redis modules (such as vector search) are outside the scope of this page.
10+
11+
## Using Valkey as a drop-in Feast online store instead of Redis
12+
13+
Make sure you have Python and `pip` installed.
14+
15+
Install the Feast SDK and CLI
16+
17+
`pip install feast`
18+
19+
In order to use Valkey as the online store, you'll need to install the redis extra:
20+
21+
`pip install 'feast[redis]'`
22+
23+
### 1. Create a feature repository
24+
25+
Bootstrap a new feature repository:
26+
27+
```
28+
feast init feast_valkey
29+
cd feast_valkey/feature_repo
30+
```
31+
32+
Update `feature_repo/feature_store.yaml` with the below contents:
33+
34+
```
35+
project: feast_valkey
36+
registry: data/registry.db
37+
provider: local
38+
online_store:
39+
type: redis
40+
connection_string: "localhost:6379"
41+
```
42+
43+
Note that the online store `type` remains `redis`: Feast talks to Valkey over the Redis protocol, and all options of the [Redis online store](redis.md) (such as `key_ttl_seconds`) apply unchanged.
44+
45+
### 2. Start Valkey
46+
47+
There are several options available to get Valkey up and running quickly. We will be using Docker for this tutorial.
48+
49+
`docker run -d -p 6379:6379 valkey/valkey:8.1`
50+
51+
### 3. Register feature definitions and deploy your feature store
52+
53+
`feast apply`
54+
55+
The `apply` command scans python files in the current directory for feature view/entity definitions, registers the objects, and deploys infrastructure.
56+
You should see the following output:
57+
58+
```
59+
....
60+
Created entity driver
61+
Created feature view driver_hourly_stats_fresh
62+
Created feature view driver_hourly_stats
63+
Created on demand feature view transformed_conv_rate
64+
Created on demand feature view transformed_conv_rate_fresh
65+
Created feature service driver_activity_v1
66+
Created feature service driver_activity_v3
67+
Created feature service driver_activity_v2
68+
```
69+
70+
## Functionality Matrix
71+
72+
The set of functionality supported by online stores is described in detail [here](overview.md#functionality).
73+
Below is a matrix indicating which functionality is supported by the Redis online store, which Feast uses to communicate with Valkey.
74+
75+
| | Redis |
76+
| :-------------------------------------------------------- | :---- |
77+
| write feature values to the online store | yes |
78+
| read feature values from the online store | yes |
79+
| update infrastructure (e.g. tables) in the online store | yes |
80+
| teardown infrastructure (e.g. tables) in the online store | yes |
81+
| generate a plan of infrastructure changes | no |
82+
| support for on-demand transforms | yes |
83+
| readable by Python SDK | yes |
84+
| readable by Java | yes |
85+
| readable by Go | yes |
86+
| support for entityless feature views | yes |
87+
| support for concurrent writing to the same key | yes |
88+
| support for ttl (time to live) at retrieval | yes |
89+
| support for deleting expired data | yes |
90+
| collocated by feature view | no |
91+
| collocated by feature service | no |
92+
| collocated by entity key | yes |
93+
94+
To compare this set of functionality against other online stores, please see the full [functionality matrix](overview.md#functionality-matrix).

pixi.lock

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

pyproject.toml

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,9 @@ dependencies = [
4343
"prometheus_client>=0.20.0,<0.25.0",
4444
"psutil",
4545
"bigtree>=0.19.2",
46-
"pyjwt",
46+
# >=2.13 for PyJWKClient's JWK-set cache surviving transient fetch
47+
# failures; the OIDC token parser reuses one client and relies on it.
48+
"pyjwt>=2.13.0",
4749
]
4850

4951
[project.optional-dependencies]

sdk/python/feast/permissions/auth/oidc_token_parser.py

Lines changed: 36 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,41 @@ def __init__(self, auth_config: OidcAuthConfig):
4141
ca_cert_path=self._auth_config.ca_cert_path,
4242
)
4343
self._k8s_auth_api = None
44+
self._jwks_client: Optional[PyJWKClient] = None # Initialize it lazily.
45+
46+
def _get_jwks_client(self) -> PyJWKClient:
47+
"""Lazily build and cache a parser-lifetime ``PyJWKClient``.
48+
49+
A per-request client starts with a cold JWK-set cache, forcing a
50+
full HTTPS fetch of the JWKS document on every authenticated
51+
request. Reusing one client lets PyJWT cache the JWK set for
52+
``jwks_cache_lifespan_seconds``, which also bounds two staleness
53+
windows: a key the IdP has removed keeps validating, and a
54+
rotation that reuses an existing ``kid`` keeps failing, for at
55+
most that long. Rotations that introduce a new ``kid`` recover
56+
immediately (``PyJWKClient.get_signing_key`` refreshes and
57+
retries once on a cache miss).
58+
"""
59+
if self._jwks_client is None:
60+
ssl_ctx = ssl.create_default_context()
61+
if not self._auth_config.verify_ssl:
62+
ssl_ctx.check_hostname = False
63+
ssl_ctx.verify_mode = ssl.CERT_NONE
64+
elif self._auth_config.ca_cert_path and os.path.exists(
65+
self._auth_config.ca_cert_path
66+
):
67+
ssl_ctx.load_verify_locations(self._auth_config.ca_cert_path)
68+
self._jwks_client = PyJWKClient(
69+
self.oidc_discovery_service.get_jwks_url(),
70+
headers={"User-agent": "custom-user-agent"},
71+
ssl_context=ssl_ctx,
72+
# Explicit so upgrades cannot silently change the staleness
73+
# window documented above, and so a hung IdP bounds how long
74+
# a fetch can block the serving path.
75+
lifespan=self._auth_config.jwks_cache_lifespan_seconds,
76+
timeout=self._auth_config.jwks_request_timeout_seconds,
77+
)
78+
return self._jwks_client
4479

4580
async def _validate_token(self, access_token: str):
4681
"""
@@ -125,21 +160,7 @@ def _decode_token(self, access_token: str) -> dict:
125160
metadata (e.g. Entra ID v1.0 tokens validated against a v2.0
126161
discovery document).
127162
"""
128-
optional_custom_headers = {"User-agent": "custom-user-agent"}
129-
ssl_ctx = ssl.create_default_context()
130-
if not self._auth_config.verify_ssl:
131-
ssl_ctx.check_hostname = False
132-
ssl_ctx.verify_mode = ssl.CERT_NONE
133-
elif self._auth_config.ca_cert_path and os.path.exists(
134-
self._auth_config.ca_cert_path
135-
):
136-
ssl_ctx.load_verify_locations(self._auth_config.ca_cert_path)
137-
jwks_client = PyJWKClient(
138-
self.oidc_discovery_service.get_jwks_url(),
139-
headers=optional_custom_headers,
140-
ssl_context=ssl_ctx,
141-
)
142-
signing_key = jwks_client.get_signing_key_from_jwt(access_token)
163+
signing_key = self._get_jwks_client().get_signing_key_from_jwt(access_token)
143164
expected_audience = self._auth_config.audience
144165
expected_issuer = self._auth_config.issuer
145166
return jwt.decode(

sdk/python/feast/permissions/auth_model.py

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
from typing import Literal, Optional, Tuple
44

5-
from pydantic import ConfigDict, model_validator
5+
from pydantic import ConfigDict, Field, model_validator
66

77
from feast.repo_config import FeastConfigBaseModel
88

@@ -47,6 +47,15 @@ class OidcAuthConfig(AuthConfig):
4747
# against a v2.0 discovery URL).
4848
audience: Optional[str] = None
4949
issuer: Optional[str] = None
50+
# How long the fetched JWK set is reused before the server refetches it.
51+
# This also bounds how long a key the IdP has revoked keeps validating
52+
# tokens, so lower it if your provider rotates or revokes aggressively;
53+
# every reduction costs a corresponding increase in JWKS fetches.
54+
jwks_cache_lifespan_seconds: int = Field(default=300, gt=0)
55+
# Network timeout for the JWKS fetch. This fetch happens inline on the
56+
# request path, so an unresponsive IdP blocks serving for at most this
57+
# long.
58+
jwks_request_timeout_seconds: float = Field(default=10, gt=0)
5059

5160

5261
class OidcClientAuthConfig(OidcAuthConfig):

0 commit comments

Comments
 (0)