Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix: Address comments
Signed-off-by: ntkathole <nikhilkathole2683@gmail.com>
  • Loading branch information
ntkathole committed Feb 27, 2026
commit dee0cd6968b67ee6e8b15e93a6555caa6a617dec
8 changes: 4 additions & 4 deletions .secrets.baseline
Original file line number Diff line number Diff line change
Expand Up @@ -934,7 +934,7 @@
"filename": "infra/feast-operator/api/v1/featurestore_types.go",
"hashed_secret": "44e17306b837162269a410204daaa5ecee4ec22c",
"is_verified": false,
"line_number": 692
"line_number": 696
}
],
"infra/feast-operator/api/v1/zz_generated.deepcopy.go": [
Expand All @@ -943,7 +943,7 @@
"filename": "infra/feast-operator/api/v1/zz_generated.deepcopy.go",
"hashed_secret": "f914fc9324de1bec1ad13dec94a8ea2ddb41fc87",
"is_verified": false,
"line_number": 658
"line_number": 663
},
{
"type": "Secret Keyword",
Expand Down Expand Up @@ -1156,7 +1156,7 @@
"filename": "infra/feast-operator/internal/controller/services/services.go",
"hashed_secret": "36dc326eb15c7bdd8d91a6b87905bcea20b637d1",
"is_verified": false,
"line_number": 178
"line_number": 173
}
],
"infra/feast-operator/internal/controller/services/tls_test.go": [
Expand Down Expand Up @@ -1539,5 +1539,5 @@
}
]
},
"generated_at": "2026-02-21T16:33:24Z"
"generated_at": "2026-02-26T14:08:35Z"
}
35 changes: 18 additions & 17 deletions docs/how-to-guides/scaling-feast.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,13 +27,17 @@ Users may also be able to build an engine to scale up materialization using exis

### Horizontal Scaling with the Feast Operator

When running Feast on Kubernetes with the [Feast Operator](./feast-on-kubernetes.md), you can horizontally scale the FeatureStore deployment by adding a `scaling` field to the `services` section of the FeatureStore CR.
When running Feast on Kubernetes with the [Feast Operator](./feast-on-kubernetes.md), you can horizontally scale the FeatureStore deployment using `spec.replicas` or HPA autoscaling. The FeatureStore CRD implements the Kubernetes [scale sub-resource](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#scale-subresource), so you can also use `kubectl scale`:

```bash
kubectl scale featurestore/my-feast --replicas=3
```

**Prerequisites:** Horizontal scaling requires **DB-backed persistence** for all enabled services (online store, offline store, and registry). File-based persistence (SQLite, DuckDB, `registry.db`) is incompatible with multiple replicas because these backends do not support concurrent access from multiple pods.

#### Static Replicas

Set a fixed number of replicas:
Set a fixed number of replicas via `spec.replicas`:

```yaml
apiVersion: feast.dev/v1
Expand All @@ -42,9 +46,8 @@ metadata:
name: sample-scaling
spec:
feastProject: my_project
replicas: 3
services:
scaling:
replicas: 3
onlineStore:
persistence:
store:
Expand All @@ -62,7 +65,7 @@ spec:

#### Autoscaling with HPA

Configure a HorizontalPodAutoscaler to dynamically scale based on metrics:
Configure a HorizontalPodAutoscaler to dynamically scale based on metrics. HPA autoscaling is configured under `services.scaling.autoscaling` and is mutually exclusive with `spec.replicas > 1`:

```yaml
apiVersion: feast.dev/v1
Expand Down Expand Up @@ -110,19 +113,19 @@ When autoscaling is configured, the operator automatically sets the deployment s
#### Validation Rules

The operator enforces the following rules:
- `replicas` and `autoscaling` are **mutually exclusive** -- you cannot set both.
- `spec.replicas > 1` and `services.scaling.autoscaling` are **mutually exclusive** -- you cannot set both.
- Scaling with `replicas > 1` or any `autoscaling` config is **rejected** if any enabled service uses file-based persistence.
- S3 (`s3://`) and GCS (`gs://`) backed registry file persistence is allowed with scaling, since these object stores support concurrent readers.

#### Using KEDA (Kubernetes Event-Driven Autoscaling)

[KEDA](https://keda.sh) is also supported as an external autoscaler. Rather than using the built-in `autoscaling` field, you can create a KEDA `ScaledObject` that targets the Feast deployment directly.
[KEDA](https://keda.sh) is also supported as an external autoscaler. KEDA should target the FeatureStore's scale sub-resource directly (since it implements the Kubernetes scale API). This is the recommended approach because the operator manages the Deployment's replica count from `spec.replicas` — targeting the Deployment directly would conflict with the operator's reconciliation.

When using KEDA, do **not** set the `scaling.autoscaling` field -- KEDA manages its own HPA. The operator will preserve the replica count set by KEDA since it does not override externally managed replicas.
When using KEDA, do **not** set `scaling.autoscaling` or `spec.replicas > 1` -- KEDA manages the replica count through the scale sub-resource.

There are a few things you must configure manually when using KEDA:
1. **Ensure DB-backed persistence** -- The CRD's CEL validation rules automatically enforce DB-backed persistence when KEDA scales `spec.replicas` above 1 via the scale sub-resource. The operator also automatically switches the deployment strategy to `RollingUpdate` when `replicas > 1`.

1. **Set the deployment strategy to `RollingUpdate`** -- The operator defaults to `Recreate` when no `scaling` config is present, which causes downtime on scale events. Override it explicitly:
2. **Configure the FeatureStore** with DB-backed persistence:

```yaml
apiVersion: feast.dev/v1
Expand All @@ -132,8 +135,6 @@ metadata:
spec:
feastProject: my_project
services:
deploymentStrategy:
type: RollingUpdate
onlineStore:
persistence:
store:
Expand All @@ -149,9 +150,7 @@ spec:
name: feast-data-stores
```

2. **Ensure DB-backed persistence** -- The operator's persistence validation only applies when the built-in `scaling` field is used. With KEDA, you are responsible for ensuring all enabled services use DB-backed persistence (not SQLite, DuckDB, or local `registry.db`).

3. **Create a KEDA `ScaledObject`** targeting the Feast deployment:
3. **Create a KEDA `ScaledObject`** targeting the FeatureStore resource:

```yaml
apiVersion: keda.sh/v1alpha1
Expand All @@ -160,8 +159,10 @@ metadata:
name: feast-scaledobject
spec:
scaleTargetRef:
name: feast-sample-keda # must match the Feast deployment name
minReplicaCount: 2
apiVersion: feast.dev/v1
kind: FeatureStore
name: sample-keda
minReplicaCount: 1
maxReplicaCount: 10
triggers:
- type: prometheus
Expand Down
25 changes: 17 additions & 8 deletions infra/feast-operator/api/v1/featurestore_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,10 @@ const (
)

// FeatureStoreSpec defines the desired state of FeatureStore
// +kubebuilder:validation:XValidation:rule="self.replicas <= 1 || !has(self.services) || !has(self.services.scaling) || !has(self.services.scaling.autoscaling)",message="replicas > 1 and services.scaling.autoscaling are mutually exclusive."
// +kubebuilder:validation:XValidation:rule="self.replicas <= 1 && (!has(self.services) || !has(self.services.scaling) || !has(self.services.scaling.autoscaling)) || (has(self.services) && has(self.services.onlineStore) && has(self.services.onlineStore.persistence) && has(self.services.onlineStore.persistence.store))",message="Scaling requires DB-backed persistence for the online store. Configure services.onlineStore.persistence.store when using replicas > 1 or autoscaling."
// +kubebuilder:validation:XValidation:rule="self.replicas <= 1 && (!has(self.services) || !has(self.services.scaling) || !has(self.services.scaling.autoscaling)) || (!has(self.services) || !has(self.services.offlineStore) || (has(self.services.offlineStore.persistence) && has(self.services.offlineStore.persistence.store)))",message="Scaling requires DB-backed persistence for the offline store. Configure services.offlineStore.persistence.store when using replicas > 1 or autoscaling."
// +kubebuilder:validation:XValidation:rule="self.replicas <= 1 && (!has(self.services) || !has(self.services.scaling) || !has(self.services.scaling.autoscaling)) || (has(self.services) && has(self.services.registry) && (has(self.services.registry.remote) || (has(self.services.registry.local) && has(self.services.registry.local.persistence) && (has(self.services.registry.local.persistence.store) || (has(self.services.registry.local.persistence.file) && has(self.services.registry.local.persistence.file.path) && (self.services.registry.local.persistence.file.path.startsWith('s3://') || self.services.registry.local.persistence.file.path.startsWith('gs://')))))))",message="Scaling requires DB-backed or remote registry. Configure registry.local.persistence.store or use a remote registry when using replicas > 1 or autoscaling. S3/GCS-backed registry is also allowed."
type FeatureStoreSpec struct {
// +kubebuilder:validation:Pattern="^[A-Za-z0-9][A-Za-z0-9_-]*$"
// FeastProject is the Feast project id. This can be any alphanumeric string with underscores and hyphens, but it cannot start with an underscore or hyphen. Required.
Expand All @@ -77,6 +81,11 @@ type FeatureStoreSpec struct {
AuthzConfig *AuthzConfig `json:"authz,omitempty"`
CronJob *FeastCronJob `json:"cronJob,omitempty"`
BatchEngine *BatchEngineConfig `json:"batchEngine,omitempty"`
// Replicas is the desired number of pod replicas. Used by the scale sub-resource.
// Mutually exclusive with services.scaling.autoscaling.
// +kubebuilder:default=1
// +kubebuilder:validation:Minimum=1
Replicas *int32 `json:"replicas"`
}

// FeastProjectDir defines how to create the feast project directory.
Expand Down Expand Up @@ -302,20 +311,15 @@ type FeatureStoreServices struct {
DisableInitContainers bool `json:"disableInitContainers,omitempty"`
// Volumes specifies the volumes to mount in the FeatureStore deployment. A corresponding `VolumeMount` should be added to whichever feast service(s) require access to said volume(s).
Volumes []corev1.Volume `json:"volumes,omitempty"`
// Scaling configures horizontal scaling for the FeatureStore deployment.
// Requires DB-based persistence for all enabled services when replicas > 1 or autoscaling is configured.
// Scaling configures horizontal scaling for the FeatureStore deployment (e.g. HPA autoscaling).
// For static replicas, use spec.replicas instead.
Scaling *ScalingConfig `json:"scaling,omitempty"`
}

// ScalingConfig configures horizontal scaling for the FeatureStore deployment.
// +kubebuilder:validation:XValidation:rule="!has(self.replicas) || !has(self.autoscaling)",message="replicas and autoscaling are mutually exclusive."
type ScalingConfig struct {
// Replicas is the static number of pod replicas. Mutually exclusive with autoscaling.
// +kubebuilder:validation:Minimum=1
// +optional
Replicas *int32 `json:"replicas,omitempty"`
// Autoscaling configures a HorizontalPodAutoscaler for the FeatureStore deployment.
// Mutually exclusive with replicas.
// Mutually exclusive with spec.replicas.
// +optional
Autoscaling *AutoscalingConfig `json:"autoscaling,omitempty"`
}
Expand Down Expand Up @@ -725,6 +729,10 @@ type FeatureStoreStatus struct {
FeastVersion string `json:"feastVersion,omitempty"`
Phase string `json:"phase,omitempty"`
ServiceHostnames ServiceHostnames `json:"serviceHostnames,omitempty"`
// Replicas is the current number of ready pod replicas (used by the scale sub-resource).
Replicas int32 `json:"replicas,omitempty"`
// Selector is the label selector for pods managed by the FeatureStore deployment (used by the scale sub-resource).
Selector string `json:"selector,omitempty"`
// ScalingStatus reports the current scaling state of the FeatureStore deployment.
ScalingStatus *ScalingStatus `json:"scalingStatus,omitempty"`
}
Expand All @@ -751,6 +759,7 @@ type ServiceHostnames struct {
// +kubebuilder:resource:shortName=feast
// +kubebuilder:printcolumn:name="Status",type=string,JSONPath=`.status.phase`
// +kubebuilder:printcolumn:name="Age",type=date,JSONPath=`.metadata.creationTimestamp`
// +kubebuilder:subresource:scale:specpath=.spec.replicas,statuspath=.status.replicas,selectorpath=.status.selector
// +kubebuilder:storageversion

// FeatureStore is the Schema for the featurestores API
Expand Down
10 changes: 5 additions & 5 deletions infra/feast-operator/api/v1/zz_generated.deepcopy.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading