Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 10 additions & 6 deletions .secrets.baseline
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,10 @@
{
"path": "detect_secrets.filters.allowlist.is_line_allowlisted"
},
{
"path": "detect_secrets.filters.common.is_baseline_file",
"filename": ".secrets.baseline"
},
{
"path": "detect_secrets.filters.common.is_ignored_due_to_verification_policies",
"min_level": 2
Expand Down Expand Up @@ -930,7 +934,7 @@
"filename": "infra/feast-operator/api/v1/featurestore_types.go",
"hashed_secret": "44e17306b837162269a410204daaa5ecee4ec22c",
"is_verified": false,
"line_number": 657
"line_number": 696
}
],
"infra/feast-operator/api/v1/zz_generated.deepcopy.go": [
Expand All @@ -939,21 +943,21 @@
"filename": "infra/feast-operator/api/v1/zz_generated.deepcopy.go",
"hashed_secret": "f914fc9324de1bec1ad13dec94a8ea2ddb41fc87",
"is_verified": false,
"line_number": 615
"line_number": 663
},
{
"type": "Secret Keyword",
"filename": "infra/feast-operator/api/v1/zz_generated.deepcopy.go",
"hashed_secret": "44e17306b837162269a410204daaa5ecee4ec22c",
"is_verified": false,
"line_number": 1123
"line_number": 1206
},
{
"type": "Secret Keyword",
"filename": "infra/feast-operator/api/v1/zz_generated.deepcopy.go",
"hashed_secret": "c2028031c154bbe86fd69bef740855c74b927dcf",
"is_verified": false,
"line_number": 1128
"line_number": 1211
}
],
"infra/feast-operator/api/v1alpha1/featurestore_types.go": [
Expand Down Expand Up @@ -1152,7 +1156,7 @@
"filename": "infra/feast-operator/internal/controller/services/services.go",
"hashed_secret": "36dc326eb15c7bdd8d91a6b87905bcea20b637d1",
"is_verified": false,
"line_number": 164
"line_number": 173
}
],
"infra/feast-operator/internal/controller/services/tls_test.go": [
Expand Down Expand Up @@ -1535,5 +1539,5 @@
}
]
},
"generated_at": "2026-02-19T06:53:49Z"
"generated_at": "2026-02-26T14:08:35Z"
}
6 changes: 4 additions & 2 deletions docs/how-to-guides/feast-on-kubernetes.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,9 @@ spec:
> _More advanced FeatureStore CR examples can be found in the feast-operator [samples directory](../../infra/feast-operator/config/samples)._

{% hint style="success" %}
Important note: Scaling a Feature Store Deployment should only be done if the configured data store(s) will support it.
**Scaling:** The Feast Operator supports horizontal scaling via static replicas, HPA autoscaling, or external autoscalers like [KEDA](https://keda.sh). Scaling requires DB-backed persistence for all enabled services.

Please check the how-to guide for some specific recommendations on [how to scale Feast](./scaling-feast.md).
See the [Horizontal Scaling with the Feast Operator](./scaling-feast.md#horizontal-scaling-with-the-feast-operator) guide for configuration details, or check the general recommendations on [how to scale Feast](./scaling-feast.md).
{% endhint %}

> _Sample scaling CRs are available at [`v1_featurestore_scaling_static.yaml`](../../infra/feast-operator/config/samples/v1_featurestore_scaling_static.yaml) and [`v1_featurestore_scaling_hpa.yaml`](../../infra/feast-operator/config/samples/v1_featurestore_scaling_hpa.yaml)._
156 changes: 155 additions & 1 deletion docs/how-to-guides/scaling-feast.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,4 +23,158 @@ However, this process does not scale for large data sets, since it's executed on
Feast supports pluggable [Compute Engines](../getting-started/components/compute-engine.md), that allow the materialization process to be scaled up.
Aside from the local process, Feast supports a [Lambda-based materialization engine](https://rtd.feast.dev/en/master/#alpha-lambda-based-engine), and a [Bytewax-based materialization engine](https://rtd.feast.dev/en/master/#bytewax-engine).

Users may also be able to build an engine to scale up materialization using existing infrastructure in their organizations.
Users may also be able to build an engine to scale up materialization using existing infrastructure in their organizations.

### Horizontal Scaling with the Feast Operator

When running Feast on Kubernetes with the [Feast Operator](./feast-on-kubernetes.md), you can horizontally scale the FeatureStore deployment using `spec.replicas` or HPA autoscaling. The FeatureStore CRD implements the Kubernetes [scale sub-resource](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#scale-subresource), so you can also use `kubectl scale`:

```bash
kubectl scale featurestore/my-feast --replicas=3
```

**Prerequisites:** Horizontal scaling requires **DB-backed persistence** for all enabled services (online store, offline store, and registry). File-based persistence (SQLite, DuckDB, `registry.db`) is incompatible with multiple replicas because these backends do not support concurrent access from multiple pods.

#### Static Replicas

Set a fixed number of replicas via `spec.replicas`:

```yaml
apiVersion: feast.dev/v1
kind: FeatureStore
metadata:
name: sample-scaling
spec:
feastProject: my_project
replicas: 3
services:
onlineStore:
persistence:
store:
type: postgres
secretRef:
name: feast-data-stores
registry:
local:
persistence:
store:
type: sql
secretRef:
name: feast-data-stores
```

#### Autoscaling with HPA

Configure a HorizontalPodAutoscaler to dynamically scale based on metrics. HPA autoscaling is configured under `services.scaling.autoscaling` and is mutually exclusive with `spec.replicas > 1`:

```yaml
apiVersion: feast.dev/v1
kind: FeatureStore
metadata:
name: sample-autoscaling
spec:
feastProject: my_project
services:
scaling:
autoscaling:
minReplicas: 2
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
onlineStore:
persistence:
store:
type: postgres
secretRef:
name: feast-data-stores
server:
resources:
requests:
cpu: 200m
memory: 256Mi
registry:
local:
persistence:
store:
type: sql
secretRef:
name: feast-data-stores
```

{% hint style="info" %}
When autoscaling is configured, the operator automatically sets the deployment strategy to `RollingUpdate` (instead of the default `Recreate`) to ensure zero-downtime scaling. You can override this by explicitly setting `deploymentStrategy` in the CR.
{% endhint %}

#### Validation Rules

The operator enforces the following rules:
- `spec.replicas > 1` and `services.scaling.autoscaling` are **mutually exclusive** -- you cannot set both.
- Scaling with `replicas > 1` or any `autoscaling` config is **rejected** if any enabled service uses file-based persistence.
- S3 (`s3://`) and GCS (`gs://`) backed registry file persistence is allowed with scaling, since these object stores support concurrent readers.

#### Using KEDA (Kubernetes Event-Driven Autoscaling)

[KEDA](https://keda.sh) is also supported as an external autoscaler. KEDA should target the FeatureStore's scale sub-resource directly (since it implements the Kubernetes scale API). This is the recommended approach because the operator manages the Deployment's replica count from `spec.replicas` — targeting the Deployment directly would conflict with the operator's reconciliation.

When using KEDA, do **not** set `scaling.autoscaling` or `spec.replicas > 1` -- KEDA manages the replica count through the scale sub-resource.

1. **Ensure DB-backed persistence** -- The CRD's CEL validation rules automatically enforce DB-backed persistence when KEDA scales `spec.replicas` above 1 via the scale sub-resource. The operator also automatically switches the deployment strategy to `RollingUpdate` when `replicas > 1`.

2. **Configure the FeatureStore** with DB-backed persistence:

```yaml
apiVersion: feast.dev/v1
kind: FeatureStore
metadata:
name: sample-keda
spec:
feastProject: my_project
services:
onlineStore:
persistence:
store:
type: postgres
secretRef:
name: feast-data-stores
registry:
local:
persistence:
store:
type: sql
secretRef:
name: feast-data-stores
```

3. **Create a KEDA `ScaledObject`** targeting the FeatureStore resource:

```yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: feast-scaledobject
spec:
scaleTargetRef:
apiVersion: feast.dev/v1
kind: FeatureStore
name: sample-keda
minReplicaCount: 1
maxReplicaCount: 10
triggers:
- type: prometheus
metadata:
serverAddress: http://prometheus.monitoring.svc:9090
metricName: http_requests_total
query: sum(rate(http_requests_total{service="feast"}[2m]))
threshold: "100"
```

{% hint style="warning" %}
KEDA-created HPAs are not owned by the Feast operator. The operator will not interfere with them, but it also will not clean them up if the FeatureStore CR is deleted. You must manage the KEDA `ScaledObject` lifecycle independently.
{% endhint %}

For the full API reference, see the [FeatureStore CRD reference](../../infra/feast-operator/docs/api/markdown/ref.md).
54 changes: 54 additions & 0 deletions infra/feast-operator/api/v1/featurestore_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ package v1

import (
appsv1 "k8s.io/api/apps/v1"
autoscalingv2 "k8s.io/api/autoscaling/v2"
batchv1 "k8s.io/api/batch/v1"
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
Expand Down Expand Up @@ -67,6 +68,10 @@ const (
)

// FeatureStoreSpec defines the desired state of FeatureStore
// +kubebuilder:validation:XValidation:rule="self.replicas <= 1 || !has(self.services) || !has(self.services.scaling) || !has(self.services.scaling.autoscaling)",message="replicas > 1 and services.scaling.autoscaling are mutually exclusive."
// +kubebuilder:validation:XValidation:rule="self.replicas <= 1 && (!has(self.services) || !has(self.services.scaling) || !has(self.services.scaling.autoscaling)) || (has(self.services) && has(self.services.onlineStore) && has(self.services.onlineStore.persistence) && has(self.services.onlineStore.persistence.store))",message="Scaling requires DB-backed persistence for the online store. Configure services.onlineStore.persistence.store when using replicas > 1 or autoscaling."
// +kubebuilder:validation:XValidation:rule="self.replicas <= 1 && (!has(self.services) || !has(self.services.scaling) || !has(self.services.scaling.autoscaling)) || (!has(self.services) || !has(self.services.offlineStore) || (has(self.services.offlineStore.persistence) && has(self.services.offlineStore.persistence.store)))",message="Scaling requires DB-backed persistence for the offline store. Configure services.offlineStore.persistence.store when using replicas > 1 or autoscaling."
// +kubebuilder:validation:XValidation:rule="self.replicas <= 1 && (!has(self.services) || !has(self.services.scaling) || !has(self.services.scaling.autoscaling)) || (has(self.services) && has(self.services.registry) && (has(self.services.registry.remote) || (has(self.services.registry.local) && has(self.services.registry.local.persistence) && (has(self.services.registry.local.persistence.store) || (has(self.services.registry.local.persistence.file) && has(self.services.registry.local.persistence.file.path) && (self.services.registry.local.persistence.file.path.startsWith('s3://') || self.services.registry.local.persistence.file.path.startsWith('gs://')))))))",message="Scaling requires DB-backed or remote registry. Configure registry.local.persistence.store or use a remote registry when using replicas > 1 or autoscaling. S3/GCS-backed registry is also allowed."
type FeatureStoreSpec struct {
// +kubebuilder:validation:Pattern="^[A-Za-z0-9][A-Za-z0-9_-]*$"
// FeastProject is the Feast project id. This can be any alphanumeric string with underscores and hyphens, but it cannot start with an underscore or hyphen. Required.
Expand All @@ -76,6 +81,11 @@ type FeatureStoreSpec struct {
AuthzConfig *AuthzConfig `json:"authz,omitempty"`
CronJob *FeastCronJob `json:"cronJob,omitempty"`
BatchEngine *BatchEngineConfig `json:"batchEngine,omitempty"`
// Replicas is the desired number of pod replicas. Used by the scale sub-resource.
// Mutually exclusive with services.scaling.autoscaling.
// +kubebuilder:default=1
// +kubebuilder:validation:Minimum=1
Replicas *int32 `json:"replicas"`
}

// FeastProjectDir defines how to create the feast project directory.
Expand Down Expand Up @@ -301,6 +311,35 @@ type FeatureStoreServices struct {
DisableInitContainers bool `json:"disableInitContainers,omitempty"`
// Volumes specifies the volumes to mount in the FeatureStore deployment. A corresponding `VolumeMount` should be added to whichever feast service(s) require access to said volume(s).
Volumes []corev1.Volume `json:"volumes,omitempty"`
// Scaling configures horizontal scaling for the FeatureStore deployment (e.g. HPA autoscaling).
// For static replicas, use spec.replicas instead.
Scaling *ScalingConfig `json:"scaling,omitempty"`
}

// ScalingConfig configures horizontal scaling for the FeatureStore deployment.
type ScalingConfig struct {
// Autoscaling configures a HorizontalPodAutoscaler for the FeatureStore deployment.
// Mutually exclusive with spec.replicas.
// +optional
Autoscaling *AutoscalingConfig `json:"autoscaling,omitempty"`
}

// AutoscalingConfig defines HPA settings for the FeatureStore deployment.
type AutoscalingConfig struct {
// MinReplicas is the lower limit for the number of replicas. Defaults to 1.
// +kubebuilder:validation:Minimum=1
// +optional
MinReplicas *int32 `json:"minReplicas,omitempty"`
// MaxReplicas is the upper limit for the number of replicas. Required.
// +kubebuilder:validation:Minimum=1
MaxReplicas int32 `json:"maxReplicas"`
// Metrics contains the specifications for which to use to calculate the desired replica count.
// If not set, defaults to 80% CPU utilization.
// +optional
Metrics []autoscalingv2.MetricSpec `json:"metrics,omitempty"`
// Behavior configures the scaling behavior of the target.
// +optional
Behavior *autoscalingv2.HorizontalPodAutoscalerBehavior `json:"behavior,omitempty"`
}

// OfflineStore configures the offline store service
Expand Down Expand Up @@ -690,6 +729,20 @@ type FeatureStoreStatus struct {
FeastVersion string `json:"feastVersion,omitempty"`
Phase string `json:"phase,omitempty"`
ServiceHostnames ServiceHostnames `json:"serviceHostnames,omitempty"`
// Replicas is the current number of ready pod replicas (used by the scale sub-resource).
Replicas int32 `json:"replicas,omitempty"`
// Selector is the label selector for pods managed by the FeatureStore deployment (used by the scale sub-resource).
Selector string `json:"selector,omitempty"`
// ScalingStatus reports the current scaling state of the FeatureStore deployment.
ScalingStatus *ScalingStatus `json:"scalingStatus,omitempty"`
}

// ScalingStatus reports the observed scaling state.
type ScalingStatus struct {
// CurrentReplicas is the current number of pod replicas.
CurrentReplicas int32 `json:"currentReplicas,omitempty"`
// DesiredReplicas is the desired number of pod replicas.
DesiredReplicas int32 `json:"desiredReplicas,omitempty"`
}

// ServiceHostnames defines the service hostnames in the format of <domain>:<port>, e.g. example.svc.cluster.local:80
Expand All @@ -706,6 +759,7 @@ type ServiceHostnames struct {
// +kubebuilder:resource:shortName=feast
// +kubebuilder:printcolumn:name="Status",type=string,JSONPath=`.status.phase`
// +kubebuilder:printcolumn:name="Age",type=date,JSONPath=`.metadata.creationTimestamp`
// +kubebuilder:subresource:scale:specpath=.spec.replicas,statuspath=.status.replicas,selectorpath=.status.selector
// +kubebuilder:storageversion

// FeatureStore is the Schema for the featurestores API
Expand Down
Loading
Loading