The observation
OidcTokenParser._decode_token constructs a fresh PyJWKClient on every call (sdk/python/feast/permissions/auth/oidc_token_parser.py):
def _decode_token(self, access_token: str) -> dict:
...
jwks_client = PyJWKClient(
self.oidc_discovery_service.get_jwks_url(), headers=..., ssl_context=ssl_ctx,
)
signing_key = jwks_client.get_signing_key_from_jwt(access_token)
PyJWKClient caches the fetched JWK set, but the cache lives on the instance, so a per-request instance starts cold every time and get_signing_key_from_jwt performs a full HTTPS round trip to the IdP's JWKS endpoint, TLS handshake included, on every authenticated request.
Reproduced on master with a mock JWKS endpoint and a hit counter: 25 authenticated decodes produce 25 JWKS fetches.
Measured impact
On a production-shaped deployment (containerized feature server, Entra ID OIDC, Redis online store):
- The JWKS fetch alone measures 124-196 ms against
login.microsoftonline.com.
- Server-side latency for single-entity online reads sat at p50 247 ms / p95 314 ms while the Redis engine idled at 0.5% CPU; the unauthenticated
/health floor on the same connection was ~50 ms. Nearly the entire request budget goes to re-fetching a document that hasn't changed.
- Because the fetch is blocking I/O in the request path, concurrent requests serialize behind it: 3 parallel readers measured p50 900 ms (an exact 3x of single-request latency), and 10 parallel readers produced 502s from a perfectly healthy process. A deployment could easily misdiagnose this as "the feature server cannot handle load" when it is one cold cache constructor.
Both the latency inflation and the concurrency collapse are the same defect.
Fix
Build the PyJWKClient lazily once per parser. The parser is constructed once per process (init_auth_manager), so PyJWT's built-in JWK-set cache (default cache_jwk_set=True, lifespan=300) then does its job. Key rotation stays safe: PyJWKClient.get_signing_key refreshes the set and retries once whenever it sees an unknown kid (verified in PyJWT 2.13, the currently resolved version - feast pins no upper bound).
After the change the same 25-decode benchmark performs exactly 1 JWKS fetch. To be precise about the concurrency half: this removes the accidental serialization behind per-request network I/O; it does not add parallelism to the server.
PR to follow shortly.
The observation
OidcTokenParser._decode_tokenconstructs a freshPyJWKClienton every call (sdk/python/feast/permissions/auth/oidc_token_parser.py):PyJWKClientcaches the fetched JWK set, but the cache lives on the instance, so a per-request instance starts cold every time andget_signing_key_from_jwtperforms a full HTTPS round trip to the IdP's JWKS endpoint, TLS handshake included, on every authenticated request.Reproduced on master with a mock JWKS endpoint and a hit counter: 25 authenticated decodes produce 25 JWKS fetches.
Measured impact
On a production-shaped deployment (containerized feature server, Entra ID OIDC, Redis online store):
login.microsoftonline.com./healthfloor on the same connection was ~50 ms. Nearly the entire request budget goes to re-fetching a document that hasn't changed.Both the latency inflation and the concurrency collapse are the same defect.
Fix
Build the
PyJWKClientlazily once per parser. The parser is constructed once per process (init_auth_manager), so PyJWT's built-in JWK-set cache (defaultcache_jwk_set=True,lifespan=300) then does its job. Key rotation stays safe:PyJWKClient.get_signing_keyrefreshes the set and retries once whenever it sees an unknownkid(verified in PyJWT 2.13, the currently resolved version - feast pins no upper bound).After the change the same 25-decode benchmark performs exactly 1 JWKS fetch. To be precise about the concurrency half: this removes the accidental serialization behind per-request network I/O; it does not add parallelism to the server.
PR to follow shortly.