Skip to content

Commit 52999f1

Browse files
VperiodtVperiodt
authored
feat: Add MLflow integration support to Feast operator (#6611)
* Feast-mlflow operator support Signed-off-by: Vanshika Vanshika <vvanshik@redhat.com> rh-pre-commit.version: 2.3.2 rh-pre-commit.check-secrets: ENABLED * add auto-discovery Signed-off-by: Vanshika Vanshika <vvanshik@redhat.com> rh-pre-commit.version: 2.3.2 rh-pre-commit.check-secrets: ENABLED * Auto-discover MLflow UI URL for Feast UI lineage links Signed-off-by: Vanshika Vanshika <vvanshik@redhat.com> rh-pre-commit.version: 2.3.2 rh-pre-commit.check-secrets: ENABLED * chore: Refresh secrets baseline for MLflow operator changes Update .secrets.baseline line numbers after MLflow CRD/controller additions so detect-secrets pre-commit passes in CI. Signed-off-by: Vanshika Vanshika <vvanshik@redhat.com> rh-pre-commit.version: 2.3.2 rh-pre-commit.check-secrets: ENABLED * chore: Sync secrets baseline with master and operator diffs Regenerate .secrets.baseline from master so detect-secrets line numbers match the MLflow operator changes and CI pre-commit stays stable. Signed-off-by: Vanshika Vanshika <vvanshik@redhat.com> rh-pre-commit.version: 2.3.2 rh-pre-commit.check-secrets: ENABLED * address reviews Signed-off-by: Vanshika Vanshika <vvanshik@redhat.com> Signed-off-by: Vanshika Vanshika <vvanshik@redhat.com> rh-pre-commit.version: 2.3.2 rh-pre-commit.check-secrets: ENABLED * fix: Use get_ui_url() in get_mlflow_feature_models for UI lineage links Use mlflow_cfg.get_ui_url() as the base URL for model links in get_mlflow_feature_models, consistent with get_mlflow_experiments. Falls back to mlflow.get_tracking_uri() when no explicit UI URL is set. Signed-off-by: Vanshika Vanshika <vvanshik@redhat.com> * add-docs Signed-off-by: Vanshika Vanshika <vvanshik@redhat.com> --------- Signed-off-by: Vanshika Vanshika <vvanshik@redhat.com>
1 parent 120a868 commit 52999f1

23 files changed

Lines changed: 1732 additions & 10 deletions

‎.secrets.baseline‎

Lines changed: 8 additions & 8 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎docs/SUMMARY.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -87,6 +87,7 @@
8787
* [5 — Security](how-to-guides/feast-operator/05-security.md)
8888
* [6 — Batch & Jobs](how-to-guides/feast-operator/06-batch-and-jobs.md)
8989
* [7 — OpenLineage & Materialization](how-to-guides/feast-operator/07-openlineage-and-materialization.md)
90+
* [8 — MLflow Integration](how-to-guides/feast-operator/08-mlflow-integration.md)
9091
* [Feast Production Deployment Topologies](how-to-guides/production-deployment-topologies.md)
9192
* [Online Server Performance Tuning](how-to-guides/online-server-performance-tuning.md)
9293
* [Customizing Feast](how-to-guides/customizing-feast/README.md)

‎docs/how-to-guides/feast-operator/08-mlflow-integration.md‎

Lines changed: 193 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,193 @@
1+
# Guide 8 — MLflow Integration
2+
3+
The operator auto-discovers MLflow on RHOAI/ODH clusters and enables experiment tracking
4+
for every FeatureStore deployment. When the MLflow operator is present and healthy, Feast
5+
pods receive MLflow configuration automatically — no manual YAML editing required.
6+
7+
---
8+
9+
## Auto-discovery
10+
11+
The operator lists all `MLflow` CRs (`mlflow.opendatahub.io/v1`) in the cluster and uses
12+
the first one with an `Available=True` or `Ready=True` condition. When found, it populates
13+
`tracking_uri` from `status.address.url` and `ui_url` from `status.url`.
14+
15+
If the MLflow CR does not report conditions (older operator versions), auto-discovery will
16+
not activate. Set `trackingUri` explicitly in that case.
17+
18+
> **No MLflow?** The FeatureStore stays Ready. Non-MLflow FeatureViews and all other Feast
19+
> services are completely unaffected.
20+
21+
---
22+
23+
## FeatureStore CR configuration
24+
25+
### Auto-enabled (default when MLflow is present)
26+
27+
No `spec.mlflow` needed. The operator auto-enables when an Available MLflow CR is detected:
28+
29+
```yaml
30+
apiVersion: feast.dev/v1
31+
kind: FeatureStore
32+
metadata:
33+
name: my-store
34+
spec:
35+
feastProject: my_project
36+
services:
37+
onlineStore: {}
38+
registry: {}
39+
ui: {}
40+
```
41+
42+
### Explicit configuration
43+
44+
Override defaults or enable additional features:
45+
46+
```yaml
47+
apiVersion: feast.dev/v1
48+
kind: FeatureStore
49+
metadata:
50+
name: my-store
51+
spec:
52+
feastProject: my_project
53+
services:
54+
onlineStore: {}
55+
registry: {}
56+
ui: {}
57+
mlflow:
58+
enabled: true
59+
trackingUri: "https://custom-mlflow.example.com:8443"
60+
uiUrl: "https://dashboard.example.com/mlflow"
61+
trackingAuth: "kubernetes-namespaced"
62+
autoLog: true
63+
autoLogEntityDf: true
64+
entityDfMaxRows: 50000
65+
logOperations: true
66+
opsExperimentSuffix: "-feast-ops"
67+
```
68+
69+
### Opt-out
70+
71+
Disable MLflow even when the MLflow operator is present:
72+
73+
```yaml
74+
spec:
75+
mlflow:
76+
enabled: false
77+
```
78+
79+
---
80+
81+
## Field reference
82+
83+
| Field | Type | Default | Description |
84+
|-------|------|---------|-------------|
85+
| `enabled` | bool | auto-detected | Master switch for MLflow integration |
86+
| `trackingUri` | string | auto-discovered | MLflow tracking server URI (in-cluster, from `status.address.url`) |
87+
| `uiUrl` | string | auto-discovered | Browser-reachable MLflow URL for Feast UI lineage links (from `status.url`) |
88+
| `trackingAuth` | *string | `"kubernetes-namespaced"` | Auth method for Feast pods calling MLflow |
89+
| `autoLog` | *bool | `true` | Auto-log feature metadata on every retrieval |
90+
| `autoLogEntityDf` | *bool | `false` | Save entity DataFrame as artifact |
91+
| `entityDfMaxRows` | *int32 | `100000` | Skip artifact for large DataFrames |
92+
| `logOperations` | *bool | `false` | Log `feast apply` / `materialize` to ops experiment |
93+
| `opsExperimentSuffix` | *string | `"-feast-ops"` | Ops experiment name suffix |
94+
| `extraConfig` | map[string]string | — | Additional YAML fields (coerced to native types) |
95+
96+
---
97+
98+
## Authentication
99+
100+
The operator injects `MLFLOW_TRACKING_AUTH` into all Feast pod containers. The MLflow
101+
Python client's auth plugin system uses this env var to attach credentials to tracking
102+
server requests.
103+
104+
| `trackingAuth` value | Behavior |
105+
|---------------------|----------|
106+
| `"kubernetes-namespaced"` (default) | SA token + `X-MLFLOW-WORKSPACE: <namespace>` header. Multi-tenant on RHOAI. |
107+
| `"kubernetes"` | SA token only. Single-tenant setups. |
108+
| `"basic"` | HTTP Basic auth via `MLFLOW_TRACKING_USERNAME` / `MLFLOW_TRACKING_PASSWORD` env vars. |
109+
| `"bearer"` | Static bearer token from `MLFLOW_TRACKING_TOKEN` env var. |
110+
| `""` (empty string) | No auth header. Local dev or unprotected MLflow. |
111+
112+
No Kubernetes RoleBinding is needed for MLflow tracking API access. The MLflow server
113+
validates the SA token directly via TokenReview.
114+
115+
---
116+
117+
## Tracking URI resolution order
118+
119+
1. Explicit `trackingUri` in the FeatureStore CR
120+
2. Auto-discovered from MLflow CR `status.address.url` (first Available/Ready CR)
121+
3. `MLFLOW_TRACKING_URI` environment variable (on workbench pods, injected by the MLflow operator)
122+
4. MLflow default (`./mlruns`)
123+
124+
---
125+
126+
## UI URL resolution order
127+
128+
Used for browser hyperlinks in Feast UI lineage panels:
129+
130+
1. Explicit `uiUrl` in the FeatureStore CR
131+
2. `MLFLOW_UI_URL` environment variable
132+
3. Auto-discovered from MLflow CR `status.url` (external gateway route)
133+
4. Falls back to `trackingUri` (works for local dev)
134+
135+
---
136+
137+
## Graceful degradation
138+
139+
| Scenario | Behavior |
140+
|----------|----------|
141+
| MLflow operator not installed | No `mlflow` block in YAML; FeatureStore stays Ready |
142+
| MLflow CR exists but not Ready | Discovery returns empty; MLflow stays off |
143+
| Tracking URI becomes unreachable | SDK logs a warning; feature retrieval is not blocked |
144+
| `spec.mlflow.enabled: false` | MLflow integration explicitly disabled |
145+
146+
---
147+
148+
## Workbench usage
149+
150+
In a RHOAI workbench notebook connected to the FeatureStore:
151+
152+
```python
153+
from feast import FeatureStore
154+
155+
store = FeatureStore(...) # from mounted client config
156+
157+
with store.mlflow.start_run(run_name="training"):
158+
df = store.get_historical_features(
159+
entity_df=entity_df,
160+
features=["driver_stats:conv_rate", "driver_stats:acc_rate"],
161+
).to_df()
162+
model = train(df)
163+
store.mlflow.log_model(model, "model")
164+
```
165+
166+
> **Dependency**: the Feast image must include `feast[mlflow]` (`mlflow` or `mlflow-skinny`).
167+
168+
---
169+
170+
## RBAC permissions
171+
172+
The operator needs `get`, `list`, `watch` on `mlflows` in the `mlflow.opendatahub.io` API
173+
group. This is included in the default operator ClusterRole.
174+
175+
```yaml
176+
- apiGroups:
177+
- mlflow.opendatahub.io
178+
resources:
179+
- mlflows
180+
verbs:
181+
- get
182+
- list
183+
- watch
184+
```
185+
186+
---
187+
188+
## See also
189+
190+
- [API field reference — `MlflowConfig`](https://github.com/feast-dev/feast/blob/stable/infra/feast-operator/docs/api/markdown/ref.md)
191+
- [MLflow DataSource reference](../../reference/mlflow.md)
192+
- [Guide 5 — Security](05-security.md) (RBAC / OIDC auth)
193+
- [Guide 7 — OpenLineage & Materialization](07-openlineage-and-materialization.md)

‎docs/how-to-guides/feast-operator/README.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@ look for store-specific YAML options in the Feast SDK docs.
3030
| 5 | [Security](05-security.md) | Kubernetes RBAC roles vs OIDC auth; TLS for all servers |
3131
| 6 | [Batch Jobs](06-batch-and-jobs.md) | `batchEngine` ConfigMap contract, `cronJob` for scheduled materialization |
3232
| 7 | [OpenLineage & Materialization](07-openlineage-and-materialization.md) | Lineage transports, API key Secret, materialization batch size |
33+
| 8 | [MLflow Integration](08-mlflow-integration.md) | Auto-discovery, experiment tracking, auth, Feast UI lineage |
3334

3435
---
3536

@@ -42,6 +43,7 @@ look for store-specific YAML options in the Feast SDK docs.
4243
- **"How do I enable Kubernetes RBAC or OIDC auth?"** → [Guide 5](05-security.md)
4344
- **"How do I schedule nightly materialization?"** → [Guide 6](06-batch-and-jobs.md)
4445
- **"How do I send lineage events to Marquez?"** → [Guide 7](07-openlineage-and-materialization.md)
46+
- **"How do I connect Feast to MLflow for experiment tracking?"** → [Guide 8](08-mlflow-integration.md)
4547
- **"What are all valid fields on `ServingConfig`?"** → [API ref](https://github.com/feast-dev/feast/blob/stable/infra/feast-operator/docs/api/markdown/ref.md#servingconfig)
4648

4749
---

‎infra/feast-operator/api/v1/featurestore_types.go‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -142,6 +142,59 @@ type OpenLineageConsumerConfig struct {
142142
NamespaceMapping map[string]string `json:"namespaceMapping,omitempty"`
143143
}
144144

145+
// MlflowConfig enables MLflow experiment tracking integration for Feast.
146+
// When enabled, feature retrieval metadata is automatically logged to MLflow runs
147+
// and the Feast UI displays lineage from feature views to registered models.
148+
// +kubebuilder:validation:XValidation:rule="!has(self.extraConfig) || !('enabled' in self.extraConfig) && !('tracking_uri' in self.extraConfig) && !('ui_url' in self.extraConfig) && !('tracking_auth' in self.extraConfig) && !('auto_log' in self.extraConfig) && !('auto_log_entity_df' in self.extraConfig) && !('entity_df_max_rows' in self.extraConfig) && !('log_operations' in self.extraConfig) && !('ops_experiment_suffix' in self.extraConfig)",message="extraConfig must not contain keys that duplicate typed fields (enabled, tracking_uri, ui_url, tracking_auth, auto_log, auto_log_entity_df, entity_df_max_rows, log_operations, ops_experiment_suffix); use the corresponding spec fields instead."
149+
type MlflowConfig struct {
150+
// Enable MLflow integration.
151+
Enabled bool `json:"enabled"`
152+
// MLflow tracking server URI. When omitted, the operator auto-discovers
153+
// from the cluster MLflow CR (status.address.url). Falls back to
154+
// MLFLOW_TRACKING_URI env var on pods.
155+
// +optional
156+
TrackingUri *string `json:"trackingUri,omitempty"`
157+
// Browser-reachable MLflow UI URL used for hyperlinks in Feast UI lineage.
158+
// When omitted, the operator auto-discovers from the MLflow CR status.url
159+
// (the external gateway route). Falls back to MLFLOW_UI_URL env var, then
160+
// to trackingUri. Only needed when the tracking URI is cluster-internal.
161+
// +optional
162+
UiUrl *string `json:"uiUrl,omitempty"`
163+
// Automatically log feature metadata on every retrieval inside an active MLflow run.
164+
// Defaults to true when enabled.
165+
// +optional
166+
AutoLog *bool `json:"autoLog,omitempty"`
167+
// Save entity DataFrame as MLflow artifact on historical retrieval.
168+
// Defaults to false.
169+
// +optional
170+
AutoLogEntityDf *bool `json:"autoLogEntityDf,omitempty"`
171+
// Maximum number of entity DataFrame rows to save as an MLflow artifact.
172+
// DataFrames exceeding this limit are skipped. Defaults to 100000.
173+
// +kubebuilder:validation:Minimum=1
174+
// +optional
175+
EntityDfMaxRows *int32 `json:"entityDfMaxRows,omitempty"`
176+
// Log feast apply and materialize operations to a separate MLflow experiment.
177+
// Defaults to false.
178+
// +optional
179+
LogOperations *bool `json:"logOperations,omitempty"`
180+
// Suffix appended to the project name for the operations experiment.
181+
// Defaults to "-feast-ops".
182+
// +optional
183+
OpsExperimentSuffix *string `json:"opsExperimentSuffix,omitempty"`
184+
// Authentication method used by Feast pods when calling the MLflow tracking
185+
// server. Common values: "kubernetes-namespaced" (token-based, default on
186+
// OpenShift AI), "basic", "bearer", or "" (no auth for local/dev).
187+
// Defaults to "kubernetes-namespaced".
188+
// +optional
189+
TrackingAuth *string `json:"trackingAuth,omitempty"`
190+
// ExtraConfig holds additional MLflow key-value settings written inline into
191+
// the mlflow block of feature_store.yaml. Boolean and integer string values
192+
// are coerced to native YAML types. Keys must be valid Feast MlflowConfig
193+
// YAML field names.
194+
// +optional
195+
ExtraConfig map[string]string `json:"extraConfig,omitempty"`
196+
}
197+
145198
// FeatureStoreSpec defines the desired state of FeatureStore
146199
// +kubebuilder:validation:XValidation:rule="self.replicas <= 1 || !has(self.services) || !has(self.services.scaling) || !has(self.services.scaling.autoscaling)",message="replicas > 1 and services.scaling.autoscaling are mutually exclusive."
147200
// +kubebuilder:validation:XValidation:rule="self.replicas <= 1 && (!has(self.services) || !has(self.services.scaling) || !has(self.services.scaling.autoscaling)) || (has(self.services) && has(self.services.onlineStore) && has(self.services.onlineStore.persistence) && has(self.services.onlineStore.persistence.store))",message="Scaling requires DB-backed persistence for the online store. Configure services.onlineStore.persistence.store when using replicas > 1 or autoscaling."
@@ -172,6 +225,12 @@ type FeatureStoreSpec struct {
172225
// Written into feature_store.yaml for all service pods.
173226
// +optional
174227
OpenLineage *OpenLineageConfig `json:"openlineage,omitempty"`
228+
// Mlflow enables MLflow experiment tracking integration for Feast.
229+
// Written into feature_store.yaml for all service pods and the client ConfigMap.
230+
// When omitted and a cluster MLflow instance is detected, defaults to enabled
231+
// with the discovered tracking URI.
232+
// +optional
233+
Mlflow *MlflowConfig `json:"mlflow,omitempty"`
175234
}
176235

177236
// FeastProjectDir defines how to create the feast project directory.

0 commit comments

Comments
 (0)