diff --git a/.secrets.baseline b/.secrets.baseline index 9f4b51f7f02..26493e56777 100644 --- a/.secrets.baseline +++ b/.secrets.baseline @@ -90,6 +90,10 @@ { "path": "detect_secrets.filters.allowlist.is_line_allowlisted" }, + { + "path": "detect_secrets.filters.common.is_baseline_file", + "filename": ".secrets.baseline" + }, { "path": "detect_secrets.filters.common.is_ignored_due_to_verification_policies", "min_level": 2 @@ -930,7 +934,7 @@ "filename": "infra/feast-operator/api/v1/featurestore_types.go", "hashed_secret": "44e17306b837162269a410204daaa5ecee4ec22c", "is_verified": false, - "line_number": 657 + "line_number": 696 } ], "infra/feast-operator/api/v1/zz_generated.deepcopy.go": [ @@ -939,21 +943,21 @@ "filename": "infra/feast-operator/api/v1/zz_generated.deepcopy.go", "hashed_secret": "f914fc9324de1bec1ad13dec94a8ea2ddb41fc87", "is_verified": false, - "line_number": 615 + "line_number": 663 }, { "type": "Secret Keyword", "filename": "infra/feast-operator/api/v1/zz_generated.deepcopy.go", "hashed_secret": "44e17306b837162269a410204daaa5ecee4ec22c", "is_verified": false, - "line_number": 1123 + "line_number": 1206 }, { "type": "Secret Keyword", "filename": "infra/feast-operator/api/v1/zz_generated.deepcopy.go", "hashed_secret": "c2028031c154bbe86fd69bef740855c74b927dcf", "is_verified": false, - "line_number": 1128 + "line_number": 1211 } ], "infra/feast-operator/api/v1alpha1/featurestore_types.go": [ @@ -1152,7 +1156,7 @@ "filename": "infra/feast-operator/internal/controller/services/services.go", "hashed_secret": "36dc326eb15c7bdd8d91a6b87905bcea20b637d1", "is_verified": false, - "line_number": 164 + "line_number": 173 } ], "infra/feast-operator/internal/controller/services/tls_test.go": [ @@ -1535,5 +1539,5 @@ } ] }, - "generated_at": "2026-02-19T06:53:49Z" + "generated_at": "2026-02-26T14:08:35Z" } diff --git a/docs/how-to-guides/feast-on-kubernetes.md b/docs/how-to-guides/feast-on-kubernetes.md index 5504dbd671a..e87e3efa8b2 100644 --- a/docs/how-to-guides/feast-on-kubernetes.md +++ b/docs/how-to-guides/feast-on-kubernetes.md @@ -65,7 +65,9 @@ spec: > _More advanced FeatureStore CR examples can be found in the feast-operator [samples directory](../../infra/feast-operator/config/samples)._ {% hint style="success" %} -Important note: Scaling a Feature Store Deployment should only be done if the configured data store(s) will support it. +**Scaling:** The Feast Operator supports horizontal scaling via static replicas, HPA autoscaling, or external autoscalers like [KEDA](https://keda.sh). Scaling requires DB-backed persistence for all enabled services. -Please check the how-to guide for some specific recommendations on [how to scale Feast](./scaling-feast.md). +See the [Horizontal Scaling with the Feast Operator](./scaling-feast.md#horizontal-scaling-with-the-feast-operator) guide for configuration details, or check the general recommendations on [how to scale Feast](./scaling-feast.md). {% endhint %} + +> _Sample scaling CRs are available at [`v1_featurestore_scaling_static.yaml`](../../infra/feast-operator/config/samples/v1_featurestore_scaling_static.yaml) and [`v1_featurestore_scaling_hpa.yaml`](../../infra/feast-operator/config/samples/v1_featurestore_scaling_hpa.yaml)._ diff --git a/docs/how-to-guides/scaling-feast.md b/docs/how-to-guides/scaling-feast.md index d0bd6aef8a0..23c13bfe5ad 100644 --- a/docs/how-to-guides/scaling-feast.md +++ b/docs/how-to-guides/scaling-feast.md @@ -23,4 +23,158 @@ However, this process does not scale for large data sets, since it's executed on Feast supports pluggable [Compute Engines](../getting-started/components/compute-engine.md), that allow the materialization process to be scaled up. Aside from the local process, Feast supports a [Lambda-based materialization engine](https://rtd.feast.dev/en/master/#alpha-lambda-based-engine), and a [Bytewax-based materialization engine](https://rtd.feast.dev/en/master/#bytewax-engine). -Users may also be able to build an engine to scale up materialization using existing infrastructure in their organizations. \ No newline at end of file +Users may also be able to build an engine to scale up materialization using existing infrastructure in their organizations. + +### Horizontal Scaling with the Feast Operator + +When running Feast on Kubernetes with the [Feast Operator](./feast-on-kubernetes.md), you can horizontally scale the FeatureStore deployment using `spec.replicas` or HPA autoscaling. The FeatureStore CRD implements the Kubernetes [scale sub-resource](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#scale-subresource), so you can also use `kubectl scale`: + +```bash +kubectl scale featurestore/my-feast --replicas=3 +``` + +**Prerequisites:** Horizontal scaling requires **DB-backed persistence** for all enabled services (online store, offline store, and registry). File-based persistence (SQLite, DuckDB, `registry.db`) is incompatible with multiple replicas because these backends do not support concurrent access from multiple pods. + +#### Static Replicas + +Set a fixed number of replicas via `spec.replicas`: + +```yaml +apiVersion: feast.dev/v1 +kind: FeatureStore +metadata: + name: sample-scaling +spec: + feastProject: my_project + replicas: 3 + services: + onlineStore: + persistence: + store: + type: postgres + secretRef: + name: feast-data-stores + registry: + local: + persistence: + store: + type: sql + secretRef: + name: feast-data-stores +``` + +#### Autoscaling with HPA + +Configure a HorizontalPodAutoscaler to dynamically scale based on metrics. HPA autoscaling is configured under `services.scaling.autoscaling` and is mutually exclusive with `spec.replicas > 1`: + +```yaml +apiVersion: feast.dev/v1 +kind: FeatureStore +metadata: + name: sample-autoscaling +spec: + feastProject: my_project + services: + scaling: + autoscaling: + minReplicas: 2 + maxReplicas: 10 + metrics: + - type: Resource + resource: + name: cpu + target: + type: Utilization + averageUtilization: 70 + onlineStore: + persistence: + store: + type: postgres + secretRef: + name: feast-data-stores + server: + resources: + requests: + cpu: 200m + memory: 256Mi + registry: + local: + persistence: + store: + type: sql + secretRef: + name: feast-data-stores +``` + +{% hint style="info" %} +When autoscaling is configured, the operator automatically sets the deployment strategy to `RollingUpdate` (instead of the default `Recreate`) to ensure zero-downtime scaling. You can override this by explicitly setting `deploymentStrategy` in the CR. +{% endhint %} + +#### Validation Rules + +The operator enforces the following rules: +- `spec.replicas > 1` and `services.scaling.autoscaling` are **mutually exclusive** -- you cannot set both. +- Scaling with `replicas > 1` or any `autoscaling` config is **rejected** if any enabled service uses file-based persistence. +- S3 (`s3://`) and GCS (`gs://`) backed registry file persistence is allowed with scaling, since these object stores support concurrent readers. + +#### Using KEDA (Kubernetes Event-Driven Autoscaling) + +[KEDA](https://keda.sh) is also supported as an external autoscaler. KEDA should target the FeatureStore's scale sub-resource directly (since it implements the Kubernetes scale API). This is the recommended approach because the operator manages the Deployment's replica count from `spec.replicas` — targeting the Deployment directly would conflict with the operator's reconciliation. + +When using KEDA, do **not** set `scaling.autoscaling` or `spec.replicas > 1` -- KEDA manages the replica count through the scale sub-resource. + +1. **Ensure DB-backed persistence** -- The CRD's CEL validation rules automatically enforce DB-backed persistence when KEDA scales `spec.replicas` above 1 via the scale sub-resource. The operator also automatically switches the deployment strategy to `RollingUpdate` when `replicas > 1`. + +2. **Configure the FeatureStore** with DB-backed persistence: + +```yaml +apiVersion: feast.dev/v1 +kind: FeatureStore +metadata: + name: sample-keda +spec: + feastProject: my_project + services: + onlineStore: + persistence: + store: + type: postgres + secretRef: + name: feast-data-stores + registry: + local: + persistence: + store: + type: sql + secretRef: + name: feast-data-stores +``` + +3. **Create a KEDA `ScaledObject`** targeting the FeatureStore resource: + +```yaml +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: feast-scaledobject +spec: + scaleTargetRef: + apiVersion: feast.dev/v1 + kind: FeatureStore + name: sample-keda + minReplicaCount: 1 + maxReplicaCount: 10 + triggers: + - type: prometheus + metadata: + serverAddress: http://prometheus.monitoring.svc:9090 + metricName: http_requests_total + query: sum(rate(http_requests_total{service="feast"}[2m])) + threshold: "100" +``` + +{% hint style="warning" %} +KEDA-created HPAs are not owned by the Feast operator. The operator will not interfere with them, but it also will not clean them up if the FeatureStore CR is deleted. You must manage the KEDA `ScaledObject` lifecycle independently. +{% endhint %} + +For the full API reference, see the [FeatureStore CRD reference](../../infra/feast-operator/docs/api/markdown/ref.md). \ No newline at end of file diff --git a/infra/feast-operator/api/v1/featurestore_types.go b/infra/feast-operator/api/v1/featurestore_types.go index 977fc586110..8928fe74ce8 100644 --- a/infra/feast-operator/api/v1/featurestore_types.go +++ b/infra/feast-operator/api/v1/featurestore_types.go @@ -18,6 +18,7 @@ package v1 import ( appsv1 "k8s.io/api/apps/v1" + autoscalingv2 "k8s.io/api/autoscaling/v2" batchv1 "k8s.io/api/batch/v1" corev1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" @@ -67,6 +68,10 @@ const ( ) // FeatureStoreSpec defines the desired state of FeatureStore +// +kubebuilder:validation:XValidation:rule="self.replicas <= 1 || !has(self.services) || !has(self.services.scaling) || !has(self.services.scaling.autoscaling)",message="replicas > 1 and services.scaling.autoscaling are mutually exclusive." +// +kubebuilder:validation:XValidation:rule="self.replicas <= 1 && (!has(self.services) || !has(self.services.scaling) || !has(self.services.scaling.autoscaling)) || (has(self.services) && has(self.services.onlineStore) && has(self.services.onlineStore.persistence) && has(self.services.onlineStore.persistence.store))",message="Scaling requires DB-backed persistence for the online store. Configure services.onlineStore.persistence.store when using replicas > 1 or autoscaling." +// +kubebuilder:validation:XValidation:rule="self.replicas <= 1 && (!has(self.services) || !has(self.services.scaling) || !has(self.services.scaling.autoscaling)) || (!has(self.services) || !has(self.services.offlineStore) || (has(self.services.offlineStore.persistence) && has(self.services.offlineStore.persistence.store)))",message="Scaling requires DB-backed persistence for the offline store. Configure services.offlineStore.persistence.store when using replicas > 1 or autoscaling." +// +kubebuilder:validation:XValidation:rule="self.replicas <= 1 && (!has(self.services) || !has(self.services.scaling) || !has(self.services.scaling.autoscaling)) || (has(self.services) && has(self.services.registry) && (has(self.services.registry.remote) || (has(self.services.registry.local) && has(self.services.registry.local.persistence) && (has(self.services.registry.local.persistence.store) || (has(self.services.registry.local.persistence.file) && has(self.services.registry.local.persistence.file.path) && (self.services.registry.local.persistence.file.path.startsWith('s3://') || self.services.registry.local.persistence.file.path.startsWith('gs://')))))))",message="Scaling requires DB-backed or remote registry. Configure registry.local.persistence.store or use a remote registry when using replicas > 1 or autoscaling. S3/GCS-backed registry is also allowed." type FeatureStoreSpec struct { // +kubebuilder:validation:Pattern="^[A-Za-z0-9][A-Za-z0-9_-]*$" // FeastProject is the Feast project id. This can be any alphanumeric string with underscores and hyphens, but it cannot start with an underscore or hyphen. Required. @@ -76,6 +81,11 @@ type FeatureStoreSpec struct { AuthzConfig *AuthzConfig `json:"authz,omitempty"` CronJob *FeastCronJob `json:"cronJob,omitempty"` BatchEngine *BatchEngineConfig `json:"batchEngine,omitempty"` + // Replicas is the desired number of pod replicas. Used by the scale sub-resource. + // Mutually exclusive with services.scaling.autoscaling. + // +kubebuilder:default=1 + // +kubebuilder:validation:Minimum=1 + Replicas *int32 `json:"replicas"` } // FeastProjectDir defines how to create the feast project directory. @@ -301,6 +311,35 @@ type FeatureStoreServices struct { DisableInitContainers bool `json:"disableInitContainers,omitempty"` // Volumes specifies the volumes to mount in the FeatureStore deployment. A corresponding `VolumeMount` should be added to whichever feast service(s) require access to said volume(s). Volumes []corev1.Volume `json:"volumes,omitempty"` + // Scaling configures horizontal scaling for the FeatureStore deployment (e.g. HPA autoscaling). + // For static replicas, use spec.replicas instead. + Scaling *ScalingConfig `json:"scaling,omitempty"` +} + +// ScalingConfig configures horizontal scaling for the FeatureStore deployment. +type ScalingConfig struct { + // Autoscaling configures a HorizontalPodAutoscaler for the FeatureStore deployment. + // Mutually exclusive with spec.replicas. + // +optional + Autoscaling *AutoscalingConfig `json:"autoscaling,omitempty"` +} + +// AutoscalingConfig defines HPA settings for the FeatureStore deployment. +type AutoscalingConfig struct { + // MinReplicas is the lower limit for the number of replicas. Defaults to 1. + // +kubebuilder:validation:Minimum=1 + // +optional + MinReplicas *int32 `json:"minReplicas,omitempty"` + // MaxReplicas is the upper limit for the number of replicas. Required. + // +kubebuilder:validation:Minimum=1 + MaxReplicas int32 `json:"maxReplicas"` + // Metrics contains the specifications for which to use to calculate the desired replica count. + // If not set, defaults to 80% CPU utilization. + // +optional + Metrics []autoscalingv2.MetricSpec `json:"metrics,omitempty"` + // Behavior configures the scaling behavior of the target. + // +optional + Behavior *autoscalingv2.HorizontalPodAutoscalerBehavior `json:"behavior,omitempty"` } // OfflineStore configures the offline store service @@ -690,6 +729,20 @@ type FeatureStoreStatus struct { FeastVersion string `json:"feastVersion,omitempty"` Phase string `json:"phase,omitempty"` ServiceHostnames ServiceHostnames `json:"serviceHostnames,omitempty"` + // Replicas is the current number of ready pod replicas (used by the scale sub-resource). + Replicas int32 `json:"replicas,omitempty"` + // Selector is the label selector for pods managed by the FeatureStore deployment (used by the scale sub-resource). + Selector string `json:"selector,omitempty"` + // ScalingStatus reports the current scaling state of the FeatureStore deployment. + ScalingStatus *ScalingStatus `json:"scalingStatus,omitempty"` +} + +// ScalingStatus reports the observed scaling state. +type ScalingStatus struct { + // CurrentReplicas is the current number of pod replicas. + CurrentReplicas int32 `json:"currentReplicas,omitempty"` + // DesiredReplicas is the desired number of pod replicas. + DesiredReplicas int32 `json:"desiredReplicas,omitempty"` } // ServiceHostnames defines the service hostnames in the format of :, e.g. example.svc.cluster.local:80 @@ -706,6 +759,7 @@ type ServiceHostnames struct { // +kubebuilder:resource:shortName=feast // +kubebuilder:printcolumn:name="Status",type=string,JSONPath=`.status.phase` // +kubebuilder:printcolumn:name="Age",type=date,JSONPath=`.metadata.creationTimestamp` +// +kubebuilder:subresource:scale:specpath=.spec.replicas,statuspath=.status.replicas,selectorpath=.status.selector // +kubebuilder:storageversion // FeatureStore is the Schema for the featurestores API diff --git a/infra/feast-operator/api/v1/zz_generated.deepcopy.go b/infra/feast-operator/api/v1/zz_generated.deepcopy.go index 870f4489a4b..6b12020435b 100644 --- a/infra/feast-operator/api/v1/zz_generated.deepcopy.go +++ b/infra/feast-operator/api/v1/zz_generated.deepcopy.go @@ -22,6 +22,7 @@ package v1 import ( appsv1 "k8s.io/api/apps/v1" + "k8s.io/api/autoscaling/v2" batchv1 "k8s.io/api/batch/v1" corev1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" @@ -53,6 +54,38 @@ func (in *AuthzConfig) DeepCopy() *AuthzConfig { return out } +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *AutoscalingConfig) DeepCopyInto(out *AutoscalingConfig) { + *out = *in + if in.MinReplicas != nil { + in, out := &in.MinReplicas, &out.MinReplicas + *out = new(int32) + **out = **in + } + if in.Metrics != nil { + in, out := &in.Metrics, &out.Metrics + *out = make([]v2.MetricSpec, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } + if in.Behavior != nil { + in, out := &in.Behavior, &out.Behavior + *out = new(v2.HorizontalPodAutoscalerBehavior) + (*in).DeepCopyInto(*out) + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new AutoscalingConfig. +func (in *AutoscalingConfig) DeepCopy() *AutoscalingConfig { + if in == nil { + return nil + } + out := new(AutoscalingConfig) + in.DeepCopyInto(out) + return out +} + // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *BatchEngineConfig) DeepCopyInto(out *BatchEngineConfig) { *out = *in @@ -342,6 +375,11 @@ func (in *FeatureStoreServices) DeepCopyInto(out *FeatureStoreServices) { (*in)[i].DeepCopyInto(&(*out)[i]) } } + if in.Scaling != nil { + in, out := &in.Scaling, &out.Scaling + *out = new(ScalingConfig) + (*in).DeepCopyInto(*out) + } } // DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new FeatureStoreServices. @@ -382,6 +420,11 @@ func (in *FeatureStoreSpec) DeepCopyInto(out *FeatureStoreSpec) { *out = new(BatchEngineConfig) (*in).DeepCopyInto(*out) } + if in.Replicas != nil { + in, out := &in.Replicas, &out.Replicas + *out = new(int32) + **out = **in + } } // DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new FeatureStoreSpec. @@ -406,6 +449,11 @@ func (in *FeatureStoreStatus) DeepCopyInto(out *FeatureStoreStatus) { } } out.ServiceHostnames = in.ServiceHostnames + if in.ScalingStatus != nil { + in, out := &in.ScalingStatus, &out.ScalingStatus + *out = new(ScalingStatus) + **out = **in + } } // DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new FeatureStoreStatus. @@ -1044,6 +1092,41 @@ func (in *RemoteRegistryConfig) DeepCopy() *RemoteRegistryConfig { return out } +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *ScalingConfig) DeepCopyInto(out *ScalingConfig) { + *out = *in + if in.Autoscaling != nil { + in, out := &in.Autoscaling, &out.Autoscaling + *out = new(AutoscalingConfig) + (*in).DeepCopyInto(*out) + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ScalingConfig. +func (in *ScalingConfig) DeepCopy() *ScalingConfig { + if in == nil { + return nil + } + out := new(ScalingConfig) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *ScalingStatus) DeepCopyInto(out *ScalingStatus) { + *out = *in +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ScalingStatus. +func (in *ScalingStatus) DeepCopy() *ScalingStatus { + if in == nil { + return nil + } + out := new(ScalingStatus) + in.DeepCopyInto(out) + return out +} + // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *SecretKeyNames) DeepCopyInto(out *SecretKeyNames) { *out = *in diff --git a/infra/feast-operator/config/crd/bases/feast.dev_featurestores.yaml b/infra/feast-operator/config/crd/bases/feast.dev_featurestores.yaml index a3acc201a1c..00a26ef5b58 100644 --- a/infra/feast-operator/config/crd/bases/feast.dev_featurestores.yaml +++ b/infra/feast-operator/config/crd/bases/feast.dev_featurestores.yaml @@ -712,6 +712,14 @@ spec: x-kubernetes-validations: - message: One selection required between init or git. rule: '[has(self.git), has(self.init)].exists_one(c, c)' + replicas: + default: 1 + description: |- + Replicas is the desired number of pod replicas. Used by the scale sub-resource. + Mutually exclusive with services. + format: int32 + minimum: 1 + type: integer services: description: FeatureStoreServices defines the desired feast services. An ephemeral onlineStore feature server is deployed by default. @@ -2353,6 +2361,578 @@ spec: x-kubernetes-validations: - message: One selection required. rule: '[has(self.local), has(self.remote)].exists_one(c, c)' + scaling: + description: Scaling configures horizontal scaling for the FeatureStore + deployment (e.g. HPA autoscaling). + properties: + autoscaling: + description: |- + Autoscaling configures a HorizontalPodAutoscaler for the FeatureStore deployment. + Mutually exclusive with spec.replicas. + properties: + behavior: + description: Behavior configures the scaling behavior + of the target. + properties: + scaleDown: + description: scaleDown is scaling policy for scaling + Down. + properties: + policies: + description: policies is a list of potential scaling + polices which can be used during scaling. + items: + description: HPAScalingPolicy is a single policy + which must hold true for a specified past + interval. + properties: + periodSeconds: + description: periodSeconds specifies the + window of time for which the policy should + hold true. + format: int32 + type: integer + type: + description: type is used to specify the + scaling policy. + type: string + value: + description: |- + value contains the amount of change which is permitted by the policy. + It must be greater than zero + format: int32 + type: integer + required: + - periodSeconds + - type + - value + type: object + type: array + x-kubernetes-list-type: atomic + selectPolicy: + description: |- + selectPolicy is used to specify which policy should be used. + If not set, the default value Max is used. + type: string + stabilizationWindowSeconds: + description: |- + stabilizationWindowSeconds is the number of seconds for which past recommendations should be + considered while scaling up + format: int32 + type: integer + type: object + scaleUp: + description: scaleUp is scaling policy for scaling + Up. + properties: + policies: + description: policies is a list of potential scaling + polices which can be used during scaling. + items: + description: HPAScalingPolicy is a single policy + which must hold true for a specified past + interval. + properties: + periodSeconds: + description: periodSeconds specifies the + window of time for which the policy should + hold true. + format: int32 + type: integer + type: + description: type is used to specify the + scaling policy. + type: string + value: + description: |- + value contains the amount of change which is permitted by the policy. + It must be greater than zero + format: int32 + type: integer + required: + - periodSeconds + - type + - value + type: object + type: array + x-kubernetes-list-type: atomic + selectPolicy: + description: |- + selectPolicy is used to specify which policy should be used. + If not set, the default value Max is used. + type: string + stabilizationWindowSeconds: + description: |- + stabilizationWindowSeconds is the number of seconds for which past recommendations should be + considered while scaling up + format: int32 + type: integer + type: object + type: object + maxReplicas: + description: MaxReplicas is the upper limit for the number + of replicas. Required. + format: int32 + minimum: 1 + type: integer + metrics: + description: Metrics contains the specifications for which + to use to calculate the desired replica count. + items: + description: |- + MetricSpec specifies how to scale based on a single metric + (only `type` and one other matching field should be set at on + properties: + containerResource: + description: |- + containerResource refers to a resource metric (such as those specified in + requests and limits) known to Kubernetes descr + properties: + container: + description: container is the name of the container + in the pods of the scaling target + type: string + name: + description: name is the name of the resource + in question. + type: string + target: + description: target specifies the target value + for the given metric + properties: + averageUtilization: + description: "averageUtilization is the + target value of the average of the\nresource + metric across all relevant pods, represented + as a " + format: int32 + type: integer + averageValue: + anyOf: + - type: integer + - type: string + description: |- + averageValue is the target value of the average of the + metric across all relevant pods (as a quantity) + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + type: + description: type represents whether the + metric type is Utilization, Value, or + AverageValue + type: string + value: + anyOf: + - type: integer + - type: string + description: value is the target value of + the metric (as a quantity). + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + required: + - type + type: object + required: + - container + - name + - target + type: object + external: + description: |- + external refers to a global metric that is not associated + with any Kubernetes object. + properties: + metric: + description: metric identifies the target metric + by name and selector + properties: + name: + description: name is the name of the given + metric + type: string + selector: + description: "selector is the string-encoded + form of a standard kubernetes label selector + for the given metric\nWhen set, it is + passed " + properties: + matchExpressions: + description: matchExpressions is a list + of label selector requirements. The + requirements are ANDed. + items: + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. + properties: + key: + description: key is the label + key that the selector applies + to. + type: string + operator: + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. + type: string + values: + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. + items: + type: string + type: array + x-kubernetes-list-type: atomic + required: + - key + - operator + type: object + type: array + x-kubernetes-list-type: atomic + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of + {key,value} pairs. + type: object + type: object + x-kubernetes-map-type: atomic + required: + - name + type: object + target: + description: target specifies the target value + for the given metric + properties: + averageUtilization: + description: "averageUtilization is the + target value of the average of the\nresource + metric across all relevant pods, represented + as a " + format: int32 + type: integer + averageValue: + anyOf: + - type: integer + - type: string + description: |- + averageValue is the target value of the average of the + metric across all relevant pods (as a quantity) + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + type: + description: type represents whether the + metric type is Utilization, Value, or + AverageValue + type: string + value: + anyOf: + - type: integer + - type: string + description: value is the target value of + the metric (as a quantity). + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + required: + - type + type: object + required: + - metric + - target + type: object + object: + description: |- + object refers to a metric describing a single kubernetes object + (for example, hits-per-second on an Ingress object). + properties: + describedObject: + description: describedObject specifies the descriptions + of a object,such as kind,name apiVersion + properties: + apiVersion: + description: apiVersion is the API version + of the referent + type: string + kind: + description: 'kind is the kind of the referent; + More info: https://git.k8s.' + type: string + name: + description: 'name is the name of the referent; + More info: https://kubernetes.' + type: string + required: + - kind + - name + type: object + metric: + description: metric identifies the target metric + by name and selector + properties: + name: + description: name is the name of the given + metric + type: string + selector: + description: "selector is the string-encoded + form of a standard kubernetes label selector + for the given metric\nWhen set, it is + passed " + properties: + matchExpressions: + description: matchExpressions is a list + of label selector requirements. The + requirements are ANDed. + items: + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. + properties: + key: + description: key is the label + key that the selector applies + to. + type: string + operator: + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. + type: string + values: + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. + items: + type: string + type: array + x-kubernetes-list-type: atomic + required: + - key + - operator + type: object + type: array + x-kubernetes-list-type: atomic + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of + {key,value} pairs. + type: object + type: object + x-kubernetes-map-type: atomic + required: + - name + type: object + target: + description: target specifies the target value + for the given metric + properties: + averageUtilization: + description: "averageUtilization is the + target value of the average of the\nresource + metric across all relevant pods, represented + as a " + format: int32 + type: integer + averageValue: + anyOf: + - type: integer + - type: string + description: |- + averageValue is the target value of the average of the + metric across all relevant pods (as a quantity) + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + type: + description: type represents whether the + metric type is Utilization, Value, or + AverageValue + type: string + value: + anyOf: + - type: integer + - type: string + description: value is the target value of + the metric (as a quantity). + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + required: + - type + type: object + required: + - describedObject + - metric + - target + type: object + pods: + description: |- + pods refers to a metric describing each pod in the current scale target + (for example, transactions-processed-per-second) + properties: + metric: + description: metric identifies the target metric + by name and selector + properties: + name: + description: name is the name of the given + metric + type: string + selector: + description: "selector is the string-encoded + form of a standard kubernetes label selector + for the given metric\nWhen set, it is + passed " + properties: + matchExpressions: + description: matchExpressions is a list + of label selector requirements. The + requirements are ANDed. + items: + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. + properties: + key: + description: key is the label + key that the selector applies + to. + type: string + operator: + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. + type: string + values: + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. + items: + type: string + type: array + x-kubernetes-list-type: atomic + required: + - key + - operator + type: object + type: array + x-kubernetes-list-type: atomic + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of + {key,value} pairs. + type: object + type: object + x-kubernetes-map-type: atomic + required: + - name + type: object + target: + description: target specifies the target value + for the given metric + properties: + averageUtilization: + description: "averageUtilization is the + target value of the average of the\nresource + metric across all relevant pods, represented + as a " + format: int32 + type: integer + averageValue: + anyOf: + - type: integer + - type: string + description: |- + averageValue is the target value of the average of the + metric across all relevant pods (as a quantity) + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + type: + description: type represents whether the + metric type is Utilization, Value, or + AverageValue + type: string + value: + anyOf: + - type: integer + - type: string + description: value is the target value of + the metric (as a quantity). + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + required: + - type + type: object + required: + - metric + - target + type: object + resource: + description: |- + resource refers to a resource metric (such as those specified in + requests and limits) known to Kubernetes describing eac + properties: + name: + description: name is the name of the resource + in question. + type: string + target: + description: target specifies the target value + for the given metric + properties: + averageUtilization: + description: "averageUtilization is the + target value of the average of the\nresource + metric across all relevant pods, represented + as a " + format: int32 + type: integer + averageValue: + anyOf: + - type: integer + - type: string + description: |- + averageValue is the target value of the average of the + metric across all relevant pods (as a quantity) + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + type: + description: type represents whether the + metric type is Utilization, Value, or + AverageValue + type: string + value: + anyOf: + - type: integer + - type: string + description: value is the target value of + the metric (as a quantity). + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + required: + - type + type: object + required: + - name + - target + type: object + type: + description: type is the type of metric source. + type: string + required: + - type + type: object + type: array + minReplicas: + description: MinReplicas is the lower limit for the number + of replicas. Defaults to 1. + format: int32 + minimum: 1 + type: integer + required: + - maxReplicas + type: object + type: object securityContext: description: PodSecurityContext holds pod-level security attributes and common container settings. @@ -4257,7 +4837,37 @@ spec: type: object required: - feastProject + - replicas type: object + x-kubernetes-validations: + - message: replicas > 1 and services.scaling.autoscaling are mutually + exclusive. + rule: self.replicas <= 1 || !has(self.services) || !has(self.services.scaling) + || !has(self.services.scaling.autoscaling) + - message: Scaling requires DB-backed persistence for the online store. + Configure services.onlineStore.persistence.store when using replicas + > 1 or autoscaling. + rule: self.replicas <= 1 && (!has(self.services) || !has(self.services.scaling) + || !has(self.services.scaling.autoscaling)) || (has(self.services) + && has(self.services.onlineStore) && has(self.services.onlineStore.persistence) + && has(self.services.onlineStore.persistence.store)) + - message: Scaling requires DB-backed persistence for the offline store. + Configure services.offlineStore.persistence.store when using replicas + > 1 or autoscaling. + rule: self.replicas <= 1 && (!has(self.services) || !has(self.services.scaling) + || !has(self.services.scaling.autoscaling)) || (!has(self.services) + || !has(self.services.offlineStore) || (has(self.services.offlineStore.persistence) + && has(self.services.offlineStore.persistence.store))) + - message: Scaling requires DB-backed or remote registry. Configure registry.local.persistence.store + or use a remote registry when using replicas > 1 or autoscaling. S3/GCS-backed + registry is also allowed. + rule: self.replicas <= 1 && (!has(self.services) || !has(self.services.scaling) + || !has(self.services.scaling.autoscaling)) || (has(self.services) + && has(self.services.registry) && (has(self.services.registry.remote) + || (has(self.services.registry.local) && has(self.services.registry.local.persistence) + && (has(self.services.registry.local.persistence.store) || (has(self.services.registry.local.persistence.file) + && has(self.services.registry.local.persistence.file.path) && (self.services.registry.local.persistence.file.path.startsWith('s3://') + || self.services.registry.local.persistence.file.path.startsWith('gs://'))))))) status: description: FeatureStoreStatus defines the observed state of FeatureStore properties: @@ -4945,6 +5555,14 @@ spec: x-kubernetes-validations: - message: One selection required between init or git. rule: '[has(self.git), has(self.init)].exists_one(c, c)' + replicas: + default: 1 + description: |- + Replicas is the desired number of pod replicas. Used by the scale sub-resource. + Mutually exclusive with services. + format: int32 + minimum: 1 + type: integer services: description: FeatureStoreServices defines the desired feast services. An ephemeral onlineStore feature server is deployed by default. @@ -6619,6 +7237,581 @@ spec: - message: One selection required. rule: '[has(self.local), has(self.remote)].exists_one(c, c)' + scaling: + description: Scaling configures horizontal scaling for the + FeatureStore deployment (e.g. HPA autoscaling). + properties: + autoscaling: + description: |- + Autoscaling configures a HorizontalPodAutoscaler for the FeatureStore deployment. + Mutually exclusive with spec.replicas. + properties: + behavior: + description: Behavior configures the scaling behavior + of the target. + properties: + scaleDown: + description: scaleDown is scaling policy for scaling + Down. + properties: + policies: + description: policies is a list of potential + scaling polices which can be used during + scaling. + items: + description: HPAScalingPolicy is a single + policy which must hold true for a specified + past interval. + properties: + periodSeconds: + description: periodSeconds specifies + the window of time for which the policy + should hold true. + format: int32 + type: integer + type: + description: type is used to specify + the scaling policy. + type: string + value: + description: |- + value contains the amount of change which is permitted by the policy. + It must be greater than zero + format: int32 + type: integer + required: + - periodSeconds + - type + - value + type: object + type: array + x-kubernetes-list-type: atomic + selectPolicy: + description: |- + selectPolicy is used to specify which policy should be used. + If not set, the default value Max is used. + type: string + stabilizationWindowSeconds: + description: |- + stabilizationWindowSeconds is the number of seconds for which past recommendations should be + considered while scaling up + format: int32 + type: integer + type: object + scaleUp: + description: scaleUp is scaling policy for scaling + Up. + properties: + policies: + description: policies is a list of potential + scaling polices which can be used during + scaling. + items: + description: HPAScalingPolicy is a single + policy which must hold true for a specified + past interval. + properties: + periodSeconds: + description: periodSeconds specifies + the window of time for which the policy + should hold true. + format: int32 + type: integer + type: + description: type is used to specify + the scaling policy. + type: string + value: + description: |- + value contains the amount of change which is permitted by the policy. + It must be greater than zero + format: int32 + type: integer + required: + - periodSeconds + - type + - value + type: object + type: array + x-kubernetes-list-type: atomic + selectPolicy: + description: |- + selectPolicy is used to specify which policy should be used. + If not set, the default value Max is used. + type: string + stabilizationWindowSeconds: + description: |- + stabilizationWindowSeconds is the number of seconds for which past recommendations should be + considered while scaling up + format: int32 + type: integer + type: object + type: object + maxReplicas: + description: MaxReplicas is the upper limit for the + number of replicas. Required. + format: int32 + minimum: 1 + type: integer + metrics: + description: Metrics contains the specifications for + which to use to calculate the desired replica count. + items: + description: |- + MetricSpec specifies how to scale based on a single metric + (only `type` and one other matching field should be set at on + properties: + containerResource: + description: |- + containerResource refers to a resource metric (such as those specified in + requests and limits) known to Kubernetes descr + properties: + container: + description: container is the name of the + container in the pods of the scaling target + type: string + name: + description: name is the name of the resource + in question. + type: string + target: + description: target specifies the target + value for the given metric + properties: + averageUtilization: + description: "averageUtilization is + the target value of the average of + the\nresource metric across all relevant + pods, represented as a " + format: int32 + type: integer + averageValue: + anyOf: + - type: integer + - type: string + description: |- + averageValue is the target value of the average of the + metric across all relevant pods (as a quantity) + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + type: + description: type represents whether + the metric type is Utilization, Value, + or AverageValue + type: string + value: + anyOf: + - type: integer + - type: string + description: value is the target value + of the metric (as a quantity). + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + required: + - type + type: object + required: + - container + - name + - target + type: object + external: + description: |- + external refers to a global metric that is not associated + with any Kubernetes object. + properties: + metric: + description: metric identifies the target + metric by name and selector + properties: + name: + description: name is the name of the + given metric + type: string + selector: + description: "selector is the string-encoded + form of a standard kubernetes label + selector for the given metric\nWhen + set, it is passed " + properties: + matchExpressions: + description: matchExpressions is + a list of label selector requirements. + The requirements are ANDed. + items: + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. + properties: + key: + description: key is the label + key that the selector applies + to. + type: string + operator: + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. + type: string + values: + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. + items: + type: string + type: array + x-kubernetes-list-type: atomic + required: + - key + - operator + type: object + type: array + x-kubernetes-list-type: atomic + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map + of {key,value} pairs. + type: object + type: object + x-kubernetes-map-type: atomic + required: + - name + type: object + target: + description: target specifies the target + value for the given metric + properties: + averageUtilization: + description: "averageUtilization is + the target value of the average of + the\nresource metric across all relevant + pods, represented as a " + format: int32 + type: integer + averageValue: + anyOf: + - type: integer + - type: string + description: |- + averageValue is the target value of the average of the + metric across all relevant pods (as a quantity) + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + type: + description: type represents whether + the metric type is Utilization, Value, + or AverageValue + type: string + value: + anyOf: + - type: integer + - type: string + description: value is the target value + of the metric (as a quantity). + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + required: + - type + type: object + required: + - metric + - target + type: object + object: + description: |- + object refers to a metric describing a single kubernetes object + (for example, hits-per-second on an Ingress object). + properties: + describedObject: + description: describedObject specifies the + descriptions of a object,such as kind,name + apiVersion + properties: + apiVersion: + description: apiVersion is the API version + of the referent + type: string + kind: + description: 'kind is the kind of the + referent; More info: https://git.k8s.' + type: string + name: + description: 'name is the name of the + referent; More info: https://kubernetes.' + type: string + required: + - kind + - name + type: object + metric: + description: metric identifies the target + metric by name and selector + properties: + name: + description: name is the name of the + given metric + type: string + selector: + description: "selector is the string-encoded + form of a standard kubernetes label + selector for the given metric\nWhen + set, it is passed " + properties: + matchExpressions: + description: matchExpressions is + a list of label selector requirements. + The requirements are ANDed. + items: + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. + properties: + key: + description: key is the label + key that the selector applies + to. + type: string + operator: + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. + type: string + values: + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. + items: + type: string + type: array + x-kubernetes-list-type: atomic + required: + - key + - operator + type: object + type: array + x-kubernetes-list-type: atomic + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map + of {key,value} pairs. + type: object + type: object + x-kubernetes-map-type: atomic + required: + - name + type: object + target: + description: target specifies the target + value for the given metric + properties: + averageUtilization: + description: "averageUtilization is + the target value of the average of + the\nresource metric across all relevant + pods, represented as a " + format: int32 + type: integer + averageValue: + anyOf: + - type: integer + - type: string + description: |- + averageValue is the target value of the average of the + metric across all relevant pods (as a quantity) + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + type: + description: type represents whether + the metric type is Utilization, Value, + or AverageValue + type: string + value: + anyOf: + - type: integer + - type: string + description: value is the target value + of the metric (as a quantity). + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + required: + - type + type: object + required: + - describedObject + - metric + - target + type: object + pods: + description: |- + pods refers to a metric describing each pod in the current scale target + (for example, transactions-processed-per-second) + properties: + metric: + description: metric identifies the target + metric by name and selector + properties: + name: + description: name is the name of the + given metric + type: string + selector: + description: "selector is the string-encoded + form of a standard kubernetes label + selector for the given metric\nWhen + set, it is passed " + properties: + matchExpressions: + description: matchExpressions is + a list of label selector requirements. + The requirements are ANDed. + items: + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. + properties: + key: + description: key is the label + key that the selector applies + to. + type: string + operator: + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. + type: string + values: + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. + items: + type: string + type: array + x-kubernetes-list-type: atomic + required: + - key + - operator + type: object + type: array + x-kubernetes-list-type: atomic + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map + of {key,value} pairs. + type: object + type: object + x-kubernetes-map-type: atomic + required: + - name + type: object + target: + description: target specifies the target + value for the given metric + properties: + averageUtilization: + description: "averageUtilization is + the target value of the average of + the\nresource metric across all relevant + pods, represented as a " + format: int32 + type: integer + averageValue: + anyOf: + - type: integer + - type: string + description: |- + averageValue is the target value of the average of the + metric across all relevant pods (as a quantity) + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + type: + description: type represents whether + the metric type is Utilization, Value, + or AverageValue + type: string + value: + anyOf: + - type: integer + - type: string + description: value is the target value + of the metric (as a quantity). + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + required: + - type + type: object + required: + - metric + - target + type: object + resource: + description: |- + resource refers to a resource metric (such as those specified in + requests and limits) known to Kubernetes describing eac + properties: + name: + description: name is the name of the resource + in question. + type: string + target: + description: target specifies the target + value for the given metric + properties: + averageUtilization: + description: "averageUtilization is + the target value of the average of + the\nresource metric across all relevant + pods, represented as a " + format: int32 + type: integer + averageValue: + anyOf: + - type: integer + - type: string + description: |- + averageValue is the target value of the average of the + metric across all relevant pods (as a quantity) + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + type: + description: type represents whether + the metric type is Utilization, Value, + or AverageValue + type: string + value: + anyOf: + - type: integer + - type: string + description: value is the target value + of the metric (as a quantity). + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + required: + - type + type: object + required: + - name + - target + type: object + type: + description: type is the type of metric source. + type: string + required: + - type + type: object + type: array + minReplicas: + description: MinReplicas is the lower limit for the + number of replicas. Defaults to 1. + format: int32 + minimum: 1 + type: integer + required: + - maxReplicas + type: object + type: object securityContext: description: PodSecurityContext holds pod-level security attributes and common container settings. @@ -8539,7 +9732,39 @@ spec: type: object required: - feastProject + - replicas type: object + x-kubernetes-validations: + - message: replicas > 1 and services.scaling.autoscaling are mutually + exclusive. + rule: self.replicas <= 1 || !has(self.services) || !has(self.services.scaling) + || !has(self.services.scaling.autoscaling) + - message: Scaling requires DB-backed persistence for the online store. + Configure services.onlineStore.persistence.store when using replicas + > 1 or autoscaling. + rule: self.replicas <= 1 && (!has(self.services) || !has(self.services.scaling) + || !has(self.services.scaling.autoscaling)) || (has(self.services) + && has(self.services.onlineStore) && has(self.services.onlineStore.persistence) + && has(self.services.onlineStore.persistence.store)) + - message: Scaling requires DB-backed persistence for the offline + store. Configure services.offlineStore.persistence.store when + using replicas > 1 or autoscaling. + rule: self.replicas <= 1 && (!has(self.services) || !has(self.services.scaling) + || !has(self.services.scaling.autoscaling)) || (!has(self.services) + || !has(self.services.offlineStore) || (has(self.services.offlineStore.persistence) + && has(self.services.offlineStore.persistence.store))) + - message: Scaling requires DB-backed or remote registry. Configure + registry.local.persistence.store or use a remote registry when + using replicas > 1 or autoscaling. S3/GCS-backed registry is also + allowed. + rule: self.replicas <= 1 && (!has(self.services) || !has(self.services.scaling) + || !has(self.services.scaling.autoscaling)) || (has(self.services) + && has(self.services.registry) && (has(self.services.registry.remote) + || (has(self.services.registry.local) && has(self.services.registry.local.persistence) + && (has(self.services.registry.local.persistence.store) || (has(self.services.registry.local.persistence.file) + && has(self.services.registry.local.persistence.file.path) && + (self.services.registry.local.persistence.file.path.startsWith('s3://') + || self.services.registry.local.persistence.file.path.startsWith('gs://'))))))) clientConfigMap: description: ConfigMap in this namespace containing a client `feature_store.yaml` for this feast deployment @@ -8604,6 +9829,28 @@ spec: type: string phase: type: string + replicas: + description: Replicas is the current number of ready pod replicas + (used by the scale sub-resource). + format: int32 + type: integer + scalingStatus: + description: ScalingStatus reports the current scaling state of the + FeatureStore deployment. + properties: + currentReplicas: + description: CurrentReplicas is the current number of pod replicas. + format: int32 + type: integer + desiredReplicas: + description: DesiredReplicas is the desired number of pod replicas. + format: int32 + type: integer + type: object + selector: + description: Selector is the label selector for pods managed by the + FeatureStore deployment (used by the scale sub-resource). + type: string serviceHostnames: description: ServiceHostnames defines the service hostnames in the format of :, e.g. example.svc.cluster.local:80 @@ -8624,6 +9871,10 @@ spec: served: true storage: true subresources: + scale: + labelSelectorPath: .status.selector + specReplicasPath: .spec.replicas + statusReplicasPath: .status.replicas status: {} - additionalPrinterColumns: - jsonPath: .status.phase diff --git a/infra/feast-operator/config/rbac/role.yaml b/infra/feast-operator/config/rbac/role.yaml index 56a952dd95c..3fa228afc6b 100644 --- a/infra/feast-operator/config/rbac/role.yaml +++ b/infra/feast-operator/config/rbac/role.yaml @@ -21,6 +21,18 @@ rules: - tokenreviews verbs: - create +- apiGroups: + - autoscaling + resources: + - horizontalpodautoscalers + verbs: + - create + - delete + - get + - list + - patch + - update + - watch - apiGroups: - batch resources: diff --git a/infra/feast-operator/config/samples/v1_featurestore_scaling_hpa.yaml b/infra/feast-operator/config/samples/v1_featurestore_scaling_hpa.yaml new file mode 100644 index 00000000000..4380b28d018 --- /dev/null +++ b/infra/feast-operator/config/samples/v1_featurestore_scaling_hpa.yaml @@ -0,0 +1,76 @@ +apiVersion: v1 +kind: Secret +metadata: + name: postgres-secret + namespace: test + labels: + app: postgres +stringData: + POSTGRES_DB: feast + POSTGRES_USER: feast + POSTGRES_PASSWORD: feast # pragma: allowlist secret +--- +apiVersion: v1 +kind: Secret +metadata: + name: feast-data-stores + namespace: test +stringData: + sql: | + path: postgresql+psycopg://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres.test.svc.cluster.local:5432/${POSTGRES_DB} + cache_ttl_seconds: 60 + sqlalchemy_config_kwargs: + echo: false + pool_pre_ping: true + postgres: | + host: postgres.test.svc.cluster.local + port: 5432 + database: ${POSTGRES_DB} + db_schema: public + user: ${POSTGRES_USER} + password: ${POSTGRES_PASSWORD} +--- +# HPA autoscaling: 2-10 replicas with DB-backed persistence +apiVersion: feast.dev/v1 +kind: FeatureStore +metadata: + name: sample-scaling-hpa + namespace: test +spec: + feastProject: my_project + services: + scaling: + autoscaling: + minReplicas: 2 + maxReplicas: 10 + metrics: + - type: Resource + resource: + name: cpu + target: + type: Utilization + averageUtilization: 70 + onlineStore: + persistence: + store: + type: postgres + secretRef: + name: feast-data-stores + server: + envFrom: + - secretRef: + name: postgres-secret + resources: + requests: + cpu: 200m + memory: 256Mi + limits: + cpu: "1" + memory: 1Gi + registry: + local: + persistence: + store: + type: sql + secretRef: + name: feast-data-stores diff --git a/infra/feast-operator/config/samples/v1_featurestore_scaling_static.yaml b/infra/feast-operator/config/samples/v1_featurestore_scaling_static.yaml new file mode 100644 index 00000000000..c0f0f21cd6a --- /dev/null +++ b/infra/feast-operator/config/samples/v1_featurestore_scaling_static.yaml @@ -0,0 +1,59 @@ +apiVersion: v1 +kind: Secret +metadata: + name: postgres-secret + namespace: test + labels: + app: postgres +stringData: + POSTGRES_DB: feast + POSTGRES_USER: feast + POSTGRES_PASSWORD: feast # pragma: allowlist secret +--- +apiVersion: v1 +kind: Secret +metadata: + name: feast-data-stores + namespace: test +stringData: + sql: | + path: postgresql+psycopg://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres.test.svc.cluster.local:5432/${POSTGRES_DB} + cache_ttl_seconds: 60 + sqlalchemy_config_kwargs: + echo: false + pool_pre_ping: true + postgres: | + host: postgres.test.svc.cluster.local + port: 5432 + database: ${POSTGRES_DB} + db_schema: public + user: ${POSTGRES_USER} + password: ${POSTGRES_PASSWORD} +--- +# Static scaling: 3 replicas with DB-backed persistence +apiVersion: feast.dev/v1 +kind: FeatureStore +metadata: + name: sample-scaling-static + namespace: test +spec: + feastProject: my_project + replicas: 3 + services: + onlineStore: + persistence: + store: + type: postgres + secretRef: + name: feast-data-stores + server: + envFrom: + - secretRef: + name: postgres-secret + registry: + local: + persistence: + store: + type: sql + secretRef: + name: feast-data-stores diff --git a/infra/feast-operator/dist/install.yaml b/infra/feast-operator/dist/install.yaml index d42a902c886..0c0b05be388 100644 --- a/infra/feast-operator/dist/install.yaml +++ b/infra/feast-operator/dist/install.yaml @@ -720,6 +720,14 @@ spec: x-kubernetes-validations: - message: One selection required between init or git. rule: '[has(self.git), has(self.init)].exists_one(c, c)' + replicas: + default: 1 + description: |- + Replicas is the desired number of pod replicas. Used by the scale sub-resource. + Mutually exclusive with services. + format: int32 + minimum: 1 + type: integer services: description: FeatureStoreServices defines the desired feast services. An ephemeral onlineStore feature server is deployed by default. @@ -2361,6 +2369,578 @@ spec: x-kubernetes-validations: - message: One selection required. rule: '[has(self.local), has(self.remote)].exists_one(c, c)' + scaling: + description: Scaling configures horizontal scaling for the FeatureStore + deployment (e.g. HPA autoscaling). + properties: + autoscaling: + description: |- + Autoscaling configures a HorizontalPodAutoscaler for the FeatureStore deployment. + Mutually exclusive with spec.replicas. + properties: + behavior: + description: Behavior configures the scaling behavior + of the target. + properties: + scaleDown: + description: scaleDown is scaling policy for scaling + Down. + properties: + policies: + description: policies is a list of potential scaling + polices which can be used during scaling. + items: + description: HPAScalingPolicy is a single policy + which must hold true for a specified past + interval. + properties: + periodSeconds: + description: periodSeconds specifies the + window of time for which the policy should + hold true. + format: int32 + type: integer + type: + description: type is used to specify the + scaling policy. + type: string + value: + description: |- + value contains the amount of change which is permitted by the policy. + It must be greater than zero + format: int32 + type: integer + required: + - periodSeconds + - type + - value + type: object + type: array + x-kubernetes-list-type: atomic + selectPolicy: + description: |- + selectPolicy is used to specify which policy should be used. + If not set, the default value Max is used. + type: string + stabilizationWindowSeconds: + description: |- + stabilizationWindowSeconds is the number of seconds for which past recommendations should be + considered while scaling up + format: int32 + type: integer + type: object + scaleUp: + description: scaleUp is scaling policy for scaling + Up. + properties: + policies: + description: policies is a list of potential scaling + polices which can be used during scaling. + items: + description: HPAScalingPolicy is a single policy + which must hold true for a specified past + interval. + properties: + periodSeconds: + description: periodSeconds specifies the + window of time for which the policy should + hold true. + format: int32 + type: integer + type: + description: type is used to specify the + scaling policy. + type: string + value: + description: |- + value contains the amount of change which is permitted by the policy. + It must be greater than zero + format: int32 + type: integer + required: + - periodSeconds + - type + - value + type: object + type: array + x-kubernetes-list-type: atomic + selectPolicy: + description: |- + selectPolicy is used to specify which policy should be used. + If not set, the default value Max is used. + type: string + stabilizationWindowSeconds: + description: |- + stabilizationWindowSeconds is the number of seconds for which past recommendations should be + considered while scaling up + format: int32 + type: integer + type: object + type: object + maxReplicas: + description: MaxReplicas is the upper limit for the number + of replicas. Required. + format: int32 + minimum: 1 + type: integer + metrics: + description: Metrics contains the specifications for which + to use to calculate the desired replica count. + items: + description: |- + MetricSpec specifies how to scale based on a single metric + (only `type` and one other matching field should be set at on + properties: + containerResource: + description: |- + containerResource refers to a resource metric (such as those specified in + requests and limits) known to Kubernetes descr + properties: + container: + description: container is the name of the container + in the pods of the scaling target + type: string + name: + description: name is the name of the resource + in question. + type: string + target: + description: target specifies the target value + for the given metric + properties: + averageUtilization: + description: "averageUtilization is the + target value of the average of the\nresource + metric across all relevant pods, represented + as a " + format: int32 + type: integer + averageValue: + anyOf: + - type: integer + - type: string + description: |- + averageValue is the target value of the average of the + metric across all relevant pods (as a quantity) + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + type: + description: type represents whether the + metric type is Utilization, Value, or + AverageValue + type: string + value: + anyOf: + - type: integer + - type: string + description: value is the target value of + the metric (as a quantity). + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + required: + - type + type: object + required: + - container + - name + - target + type: object + external: + description: |- + external refers to a global metric that is not associated + with any Kubernetes object. + properties: + metric: + description: metric identifies the target metric + by name and selector + properties: + name: + description: name is the name of the given + metric + type: string + selector: + description: "selector is the string-encoded + form of a standard kubernetes label selector + for the given metric\nWhen set, it is + passed " + properties: + matchExpressions: + description: matchExpressions is a list + of label selector requirements. The + requirements are ANDed. + items: + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. + properties: + key: + description: key is the label + key that the selector applies + to. + type: string + operator: + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. + type: string + values: + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. + items: + type: string + type: array + x-kubernetes-list-type: atomic + required: + - key + - operator + type: object + type: array + x-kubernetes-list-type: atomic + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of + {key,value} pairs. + type: object + type: object + x-kubernetes-map-type: atomic + required: + - name + type: object + target: + description: target specifies the target value + for the given metric + properties: + averageUtilization: + description: "averageUtilization is the + target value of the average of the\nresource + metric across all relevant pods, represented + as a " + format: int32 + type: integer + averageValue: + anyOf: + - type: integer + - type: string + description: |- + averageValue is the target value of the average of the + metric across all relevant pods (as a quantity) + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + type: + description: type represents whether the + metric type is Utilization, Value, or + AverageValue + type: string + value: + anyOf: + - type: integer + - type: string + description: value is the target value of + the metric (as a quantity). + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + required: + - type + type: object + required: + - metric + - target + type: object + object: + description: |- + object refers to a metric describing a single kubernetes object + (for example, hits-per-second on an Ingress object). + properties: + describedObject: + description: describedObject specifies the descriptions + of a object,such as kind,name apiVersion + properties: + apiVersion: + description: apiVersion is the API version + of the referent + type: string + kind: + description: 'kind is the kind of the referent; + More info: https://git.k8s.' + type: string + name: + description: 'name is the name of the referent; + More info: https://kubernetes.' + type: string + required: + - kind + - name + type: object + metric: + description: metric identifies the target metric + by name and selector + properties: + name: + description: name is the name of the given + metric + type: string + selector: + description: "selector is the string-encoded + form of a standard kubernetes label selector + for the given metric\nWhen set, it is + passed " + properties: + matchExpressions: + description: matchExpressions is a list + of label selector requirements. The + requirements are ANDed. + items: + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. + properties: + key: + description: key is the label + key that the selector applies + to. + type: string + operator: + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. + type: string + values: + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. + items: + type: string + type: array + x-kubernetes-list-type: atomic + required: + - key + - operator + type: object + type: array + x-kubernetes-list-type: atomic + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of + {key,value} pairs. + type: object + type: object + x-kubernetes-map-type: atomic + required: + - name + type: object + target: + description: target specifies the target value + for the given metric + properties: + averageUtilization: + description: "averageUtilization is the + target value of the average of the\nresource + metric across all relevant pods, represented + as a " + format: int32 + type: integer + averageValue: + anyOf: + - type: integer + - type: string + description: |- + averageValue is the target value of the average of the + metric across all relevant pods (as a quantity) + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + type: + description: type represents whether the + metric type is Utilization, Value, or + AverageValue + type: string + value: + anyOf: + - type: integer + - type: string + description: value is the target value of + the metric (as a quantity). + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + required: + - type + type: object + required: + - describedObject + - metric + - target + type: object + pods: + description: |- + pods refers to a metric describing each pod in the current scale target + (for example, transactions-processed-per-second) + properties: + metric: + description: metric identifies the target metric + by name and selector + properties: + name: + description: name is the name of the given + metric + type: string + selector: + description: "selector is the string-encoded + form of a standard kubernetes label selector + for the given metric\nWhen set, it is + passed " + properties: + matchExpressions: + description: matchExpressions is a list + of label selector requirements. The + requirements are ANDed. + items: + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. + properties: + key: + description: key is the label + key that the selector applies + to. + type: string + operator: + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. + type: string + values: + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. + items: + type: string + type: array + x-kubernetes-list-type: atomic + required: + - key + - operator + type: object + type: array + x-kubernetes-list-type: atomic + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of + {key,value} pairs. + type: object + type: object + x-kubernetes-map-type: atomic + required: + - name + type: object + target: + description: target specifies the target value + for the given metric + properties: + averageUtilization: + description: "averageUtilization is the + target value of the average of the\nresource + metric across all relevant pods, represented + as a " + format: int32 + type: integer + averageValue: + anyOf: + - type: integer + - type: string + description: |- + averageValue is the target value of the average of the + metric across all relevant pods (as a quantity) + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + type: + description: type represents whether the + metric type is Utilization, Value, or + AverageValue + type: string + value: + anyOf: + - type: integer + - type: string + description: value is the target value of + the metric (as a quantity). + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + required: + - type + type: object + required: + - metric + - target + type: object + resource: + description: |- + resource refers to a resource metric (such as those specified in + requests and limits) known to Kubernetes describing eac + properties: + name: + description: name is the name of the resource + in question. + type: string + target: + description: target specifies the target value + for the given metric + properties: + averageUtilization: + description: "averageUtilization is the + target value of the average of the\nresource + metric across all relevant pods, represented + as a " + format: int32 + type: integer + averageValue: + anyOf: + - type: integer + - type: string + description: |- + averageValue is the target value of the average of the + metric across all relevant pods (as a quantity) + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + type: + description: type represents whether the + metric type is Utilization, Value, or + AverageValue + type: string + value: + anyOf: + - type: integer + - type: string + description: value is the target value of + the metric (as a quantity). + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + required: + - type + type: object + required: + - name + - target + type: object + type: + description: type is the type of metric source. + type: string + required: + - type + type: object + type: array + minReplicas: + description: MinReplicas is the lower limit for the number + of replicas. Defaults to 1. + format: int32 + minimum: 1 + type: integer + required: + - maxReplicas + type: object + type: object securityContext: description: PodSecurityContext holds pod-level security attributes and common container settings. @@ -4265,7 +4845,37 @@ spec: type: object required: - feastProject + - replicas type: object + x-kubernetes-validations: + - message: replicas > 1 and services.scaling.autoscaling are mutually + exclusive. + rule: self.replicas <= 1 || !has(self.services) || !has(self.services.scaling) + || !has(self.services.scaling.autoscaling) + - message: Scaling requires DB-backed persistence for the online store. + Configure services.onlineStore.persistence.store when using replicas + > 1 or autoscaling. + rule: self.replicas <= 1 && (!has(self.services) || !has(self.services.scaling) + || !has(self.services.scaling.autoscaling)) || (has(self.services) + && has(self.services.onlineStore) && has(self.services.onlineStore.persistence) + && has(self.services.onlineStore.persistence.store)) + - message: Scaling requires DB-backed persistence for the offline store. + Configure services.offlineStore.persistence.store when using replicas + > 1 or autoscaling. + rule: self.replicas <= 1 && (!has(self.services) || !has(self.services.scaling) + || !has(self.services.scaling.autoscaling)) || (!has(self.services) + || !has(self.services.offlineStore) || (has(self.services.offlineStore.persistence) + && has(self.services.offlineStore.persistence.store))) + - message: Scaling requires DB-backed or remote registry. Configure registry.local.persistence.store + or use a remote registry when using replicas > 1 or autoscaling. S3/GCS-backed + registry is also allowed. + rule: self.replicas <= 1 && (!has(self.services) || !has(self.services.scaling) + || !has(self.services.scaling.autoscaling)) || (has(self.services) + && has(self.services.registry) && (has(self.services.registry.remote) + || (has(self.services.registry.local) && has(self.services.registry.local.persistence) + && (has(self.services.registry.local.persistence.store) || (has(self.services.registry.local.persistence.file) + && has(self.services.registry.local.persistence.file.path) && (self.services.registry.local.persistence.file.path.startsWith('s3://') + || self.services.registry.local.persistence.file.path.startsWith('gs://'))))))) status: description: FeatureStoreStatus defines the observed state of FeatureStore properties: @@ -4953,6 +5563,14 @@ spec: x-kubernetes-validations: - message: One selection required between init or git. rule: '[has(self.git), has(self.init)].exists_one(c, c)' + replicas: + default: 1 + description: |- + Replicas is the desired number of pod replicas. Used by the scale sub-resource. + Mutually exclusive with services. + format: int32 + minimum: 1 + type: integer services: description: FeatureStoreServices defines the desired feast services. An ephemeral onlineStore feature server is deployed by default. @@ -6627,6 +7245,581 @@ spec: - message: One selection required. rule: '[has(self.local), has(self.remote)].exists_one(c, c)' + scaling: + description: Scaling configures horizontal scaling for the + FeatureStore deployment (e.g. HPA autoscaling). + properties: + autoscaling: + description: |- + Autoscaling configures a HorizontalPodAutoscaler for the FeatureStore deployment. + Mutually exclusive with spec.replicas. + properties: + behavior: + description: Behavior configures the scaling behavior + of the target. + properties: + scaleDown: + description: scaleDown is scaling policy for scaling + Down. + properties: + policies: + description: policies is a list of potential + scaling polices which can be used during + scaling. + items: + description: HPAScalingPolicy is a single + policy which must hold true for a specified + past interval. + properties: + periodSeconds: + description: periodSeconds specifies + the window of time for which the policy + should hold true. + format: int32 + type: integer + type: + description: type is used to specify + the scaling policy. + type: string + value: + description: |- + value contains the amount of change which is permitted by the policy. + It must be greater than zero + format: int32 + type: integer + required: + - periodSeconds + - type + - value + type: object + type: array + x-kubernetes-list-type: atomic + selectPolicy: + description: |- + selectPolicy is used to specify which policy should be used. + If not set, the default value Max is used. + type: string + stabilizationWindowSeconds: + description: |- + stabilizationWindowSeconds is the number of seconds for which past recommendations should be + considered while scaling up + format: int32 + type: integer + type: object + scaleUp: + description: scaleUp is scaling policy for scaling + Up. + properties: + policies: + description: policies is a list of potential + scaling polices which can be used during + scaling. + items: + description: HPAScalingPolicy is a single + policy which must hold true for a specified + past interval. + properties: + periodSeconds: + description: periodSeconds specifies + the window of time for which the policy + should hold true. + format: int32 + type: integer + type: + description: type is used to specify + the scaling policy. + type: string + value: + description: |- + value contains the amount of change which is permitted by the policy. + It must be greater than zero + format: int32 + type: integer + required: + - periodSeconds + - type + - value + type: object + type: array + x-kubernetes-list-type: atomic + selectPolicy: + description: |- + selectPolicy is used to specify which policy should be used. + If not set, the default value Max is used. + type: string + stabilizationWindowSeconds: + description: |- + stabilizationWindowSeconds is the number of seconds for which past recommendations should be + considered while scaling up + format: int32 + type: integer + type: object + type: object + maxReplicas: + description: MaxReplicas is the upper limit for the + number of replicas. Required. + format: int32 + minimum: 1 + type: integer + metrics: + description: Metrics contains the specifications for + which to use to calculate the desired replica count. + items: + description: |- + MetricSpec specifies how to scale based on a single metric + (only `type` and one other matching field should be set at on + properties: + containerResource: + description: |- + containerResource refers to a resource metric (such as those specified in + requests and limits) known to Kubernetes descr + properties: + container: + description: container is the name of the + container in the pods of the scaling target + type: string + name: + description: name is the name of the resource + in question. + type: string + target: + description: target specifies the target + value for the given metric + properties: + averageUtilization: + description: "averageUtilization is + the target value of the average of + the\nresource metric across all relevant + pods, represented as a " + format: int32 + type: integer + averageValue: + anyOf: + - type: integer + - type: string + description: |- + averageValue is the target value of the average of the + metric across all relevant pods (as a quantity) + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + type: + description: type represents whether + the metric type is Utilization, Value, + or AverageValue + type: string + value: + anyOf: + - type: integer + - type: string + description: value is the target value + of the metric (as a quantity). + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + required: + - type + type: object + required: + - container + - name + - target + type: object + external: + description: |- + external refers to a global metric that is not associated + with any Kubernetes object. + properties: + metric: + description: metric identifies the target + metric by name and selector + properties: + name: + description: name is the name of the + given metric + type: string + selector: + description: "selector is the string-encoded + form of a standard kubernetes label + selector for the given metric\nWhen + set, it is passed " + properties: + matchExpressions: + description: matchExpressions is + a list of label selector requirements. + The requirements are ANDed. + items: + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. + properties: + key: + description: key is the label + key that the selector applies + to. + type: string + operator: + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. + type: string + values: + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. + items: + type: string + type: array + x-kubernetes-list-type: atomic + required: + - key + - operator + type: object + type: array + x-kubernetes-list-type: atomic + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map + of {key,value} pairs. + type: object + type: object + x-kubernetes-map-type: atomic + required: + - name + type: object + target: + description: target specifies the target + value for the given metric + properties: + averageUtilization: + description: "averageUtilization is + the target value of the average of + the\nresource metric across all relevant + pods, represented as a " + format: int32 + type: integer + averageValue: + anyOf: + - type: integer + - type: string + description: |- + averageValue is the target value of the average of the + metric across all relevant pods (as a quantity) + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + type: + description: type represents whether + the metric type is Utilization, Value, + or AverageValue + type: string + value: + anyOf: + - type: integer + - type: string + description: value is the target value + of the metric (as a quantity). + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + required: + - type + type: object + required: + - metric + - target + type: object + object: + description: |- + object refers to a metric describing a single kubernetes object + (for example, hits-per-second on an Ingress object). + properties: + describedObject: + description: describedObject specifies the + descriptions of a object,such as kind,name + apiVersion + properties: + apiVersion: + description: apiVersion is the API version + of the referent + type: string + kind: + description: 'kind is the kind of the + referent; More info: https://git.k8s.' + type: string + name: + description: 'name is the name of the + referent; More info: https://kubernetes.' + type: string + required: + - kind + - name + type: object + metric: + description: metric identifies the target + metric by name and selector + properties: + name: + description: name is the name of the + given metric + type: string + selector: + description: "selector is the string-encoded + form of a standard kubernetes label + selector for the given metric\nWhen + set, it is passed " + properties: + matchExpressions: + description: matchExpressions is + a list of label selector requirements. + The requirements are ANDed. + items: + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. + properties: + key: + description: key is the label + key that the selector applies + to. + type: string + operator: + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. + type: string + values: + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. + items: + type: string + type: array + x-kubernetes-list-type: atomic + required: + - key + - operator + type: object + type: array + x-kubernetes-list-type: atomic + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map + of {key,value} pairs. + type: object + type: object + x-kubernetes-map-type: atomic + required: + - name + type: object + target: + description: target specifies the target + value for the given metric + properties: + averageUtilization: + description: "averageUtilization is + the target value of the average of + the\nresource metric across all relevant + pods, represented as a " + format: int32 + type: integer + averageValue: + anyOf: + - type: integer + - type: string + description: |- + averageValue is the target value of the average of the + metric across all relevant pods (as a quantity) + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + type: + description: type represents whether + the metric type is Utilization, Value, + or AverageValue + type: string + value: + anyOf: + - type: integer + - type: string + description: value is the target value + of the metric (as a quantity). + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + required: + - type + type: object + required: + - describedObject + - metric + - target + type: object + pods: + description: |- + pods refers to a metric describing each pod in the current scale target + (for example, transactions-processed-per-second) + properties: + metric: + description: metric identifies the target + metric by name and selector + properties: + name: + description: name is the name of the + given metric + type: string + selector: + description: "selector is the string-encoded + form of a standard kubernetes label + selector for the given metric\nWhen + set, it is passed " + properties: + matchExpressions: + description: matchExpressions is + a list of label selector requirements. + The requirements are ANDed. + items: + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. + properties: + key: + description: key is the label + key that the selector applies + to. + type: string + operator: + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. + type: string + values: + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. + items: + type: string + type: array + x-kubernetes-list-type: atomic + required: + - key + - operator + type: object + type: array + x-kubernetes-list-type: atomic + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map + of {key,value} pairs. + type: object + type: object + x-kubernetes-map-type: atomic + required: + - name + type: object + target: + description: target specifies the target + value for the given metric + properties: + averageUtilization: + description: "averageUtilization is + the target value of the average of + the\nresource metric across all relevant + pods, represented as a " + format: int32 + type: integer + averageValue: + anyOf: + - type: integer + - type: string + description: |- + averageValue is the target value of the average of the + metric across all relevant pods (as a quantity) + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + type: + description: type represents whether + the metric type is Utilization, Value, + or AverageValue + type: string + value: + anyOf: + - type: integer + - type: string + description: value is the target value + of the metric (as a quantity). + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + required: + - type + type: object + required: + - metric + - target + type: object + resource: + description: |- + resource refers to a resource metric (such as those specified in + requests and limits) known to Kubernetes describing eac + properties: + name: + description: name is the name of the resource + in question. + type: string + target: + description: target specifies the target + value for the given metric + properties: + averageUtilization: + description: "averageUtilization is + the target value of the average of + the\nresource metric across all relevant + pods, represented as a " + format: int32 + type: integer + averageValue: + anyOf: + - type: integer + - type: string + description: |- + averageValue is the target value of the average of the + metric across all relevant pods (as a quantity) + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + type: + description: type represents whether + the metric type is Utilization, Value, + or AverageValue + type: string + value: + anyOf: + - type: integer + - type: string + description: value is the target value + of the metric (as a quantity). + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + required: + - type + type: object + required: + - name + - target + type: object + type: + description: type is the type of metric source. + type: string + required: + - type + type: object + type: array + minReplicas: + description: MinReplicas is the lower limit for the + number of replicas. Defaults to 1. + format: int32 + minimum: 1 + type: integer + required: + - maxReplicas + type: object + type: object securityContext: description: PodSecurityContext holds pod-level security attributes and common container settings. @@ -8547,7 +9740,39 @@ spec: type: object required: - feastProject + - replicas type: object + x-kubernetes-validations: + - message: replicas > 1 and services.scaling.autoscaling are mutually + exclusive. + rule: self.replicas <= 1 || !has(self.services) || !has(self.services.scaling) + || !has(self.services.scaling.autoscaling) + - message: Scaling requires DB-backed persistence for the online store. + Configure services.onlineStore.persistence.store when using replicas + > 1 or autoscaling. + rule: self.replicas <= 1 && (!has(self.services) || !has(self.services.scaling) + || !has(self.services.scaling.autoscaling)) || (has(self.services) + && has(self.services.onlineStore) && has(self.services.onlineStore.persistence) + && has(self.services.onlineStore.persistence.store)) + - message: Scaling requires DB-backed persistence for the offline + store. Configure services.offlineStore.persistence.store when + using replicas > 1 or autoscaling. + rule: self.replicas <= 1 && (!has(self.services) || !has(self.services.scaling) + || !has(self.services.scaling.autoscaling)) || (!has(self.services) + || !has(self.services.offlineStore) || (has(self.services.offlineStore.persistence) + && has(self.services.offlineStore.persistence.store))) + - message: Scaling requires DB-backed or remote registry. Configure + registry.local.persistence.store or use a remote registry when + using replicas > 1 or autoscaling. S3/GCS-backed registry is also + allowed. + rule: self.replicas <= 1 && (!has(self.services) || !has(self.services.scaling) + || !has(self.services.scaling.autoscaling)) || (has(self.services) + && has(self.services.registry) && (has(self.services.registry.remote) + || (has(self.services.registry.local) && has(self.services.registry.local.persistence) + && (has(self.services.registry.local.persistence.store) || (has(self.services.registry.local.persistence.file) + && has(self.services.registry.local.persistence.file.path) && + (self.services.registry.local.persistence.file.path.startsWith('s3://') + || self.services.registry.local.persistence.file.path.startsWith('gs://'))))))) clientConfigMap: description: ConfigMap in this namespace containing a client `feature_store.yaml` for this feast deployment @@ -8612,6 +9837,28 @@ spec: type: string phase: type: string + replicas: + description: Replicas is the current number of ready pod replicas + (used by the scale sub-resource). + format: int32 + type: integer + scalingStatus: + description: ScalingStatus reports the current scaling state of the + FeatureStore deployment. + properties: + currentReplicas: + description: CurrentReplicas is the current number of pod replicas. + format: int32 + type: integer + desiredReplicas: + description: DesiredReplicas is the desired number of pod replicas. + format: int32 + type: integer + type: object + selector: + description: Selector is the label selector for pods managed by the + FeatureStore deployment (used by the scale sub-resource). + type: string serviceHostnames: description: ServiceHostnames defines the service hostnames in the format of :, e.g. example.svc.cluster.local:80 @@ -8632,6 +9879,10 @@ spec: served: true storage: true subresources: + scale: + labelSelectorPath: .status.selector + specReplicasPath: .spec.replicas + statusReplicasPath: .status.replicas status: {} - additionalPrinterColumns: - jsonPath: .status.phase @@ -17325,6 +18576,18 @@ rules: - tokenreviews verbs: - create +- apiGroups: + - autoscaling + resources: + - horizontalpodautoscalers + verbs: + - create + - delete + - get + - list + - patch + - update + - watch - apiGroups: - batch resources: diff --git a/infra/feast-operator/docs/api/markdown/ref.md b/infra/feast-operator/docs/api/markdown/ref.md index ce64e4dd3ec..1e2367a583f 100644 --- a/infra/feast-operator/docs/api/markdown/ref.md +++ b/infra/feast-operator/docs/api/markdown/ref.md @@ -28,6 +28,24 @@ _Appears in:_ | `oidc` _[OidcAuthz](#oidcauthz)_ | | +#### AutoscalingConfig + + + +AutoscalingConfig defines HPA settings for the FeatureStore deployment. + +_Appears in:_ +- [ScalingConfig](#scalingconfig) + +| Field | Description | +| --- | --- | +| `minReplicas` _integer_ | MinReplicas is the lower limit for the number of replicas. Defaults to 1. | +| `maxReplicas` _integer_ | MaxReplicas is the upper limit for the number of replicas. Required. | +| `metrics` _[MetricSpec](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#metricspec-v2-autoscaling) array_ | Metrics contains the specifications for which to use to calculate the desired replica count. +If not set, defaults to 80% CPU utilization. | +| `behavior` _[HorizontalPodAutoscalerBehavior](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#horizontalpodautoscalerbehavior-v2-autoscaling)_ | Behavior configures the scaling behavior of the target. | + + #### BatchEngineConfig @@ -223,6 +241,8 @@ _Appears in:_ | `securityContext` _[PodSecurityContext](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#podsecuritycontext-v1-core)_ | | | `disableInitContainers` _boolean_ | Disable the 'feast repo initialization' initContainer | | `volumes` _[Volume](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#volume-v1-core) array_ | Volumes specifies the volumes to mount in the FeatureStore deployment. A corresponding `VolumeMount` should be added to whichever feast service(s) require access to said volume(s). | +| `scaling` _[ScalingConfig](#scalingconfig)_ | Scaling configures horizontal scaling for the FeatureStore deployment (e.g. HPA autoscaling). +For static replicas, use spec.replicas instead. | #### FeatureStoreSpec @@ -243,6 +263,8 @@ _Appears in:_ | `authz` _[AuthzConfig](#authzconfig)_ | | | `cronJob` _[FeastCronJob](#feastcronjob)_ | | | `batchEngine` _[BatchEngineConfig](#batchengineconfig)_ | | +| `replicas` _integer_ | Replicas is the desired number of pod replicas. Used by the scale sub-resource. +Mutually exclusive with services.scaling.autoscaling. | #### FeatureStoreStatus @@ -263,6 +285,9 @@ _Appears in:_ | `feastVersion` _string_ | | | `phase` _string_ | | | `serviceHostnames` _[ServiceHostnames](#servicehostnames)_ | | +| `replicas` _integer_ | Replicas is the current number of ready pod replicas (used by the scale sub-resource). | +| `selector` _string_ | Selector is the label selector for pods managed by the FeatureStore deployment (used by the scale sub-resource). | +| `scalingStatus` _[ScalingStatus](#scalingstatus)_ | ScalingStatus reports the current scaling state of the FeatureStore deployment. | #### GitCloneOptions @@ -747,6 +772,36 @@ _Appears in:_ | `tls` _[TlsRemoteRegistryConfigs](#tlsremoteregistryconfigs)_ | | +#### ScalingConfig + + + +ScalingConfig configures horizontal scaling for the FeatureStore deployment. + +_Appears in:_ +- [FeatureStoreServices](#featurestoreservices) + +| Field | Description | +| --- | --- | +| `autoscaling` _[AutoscalingConfig](#autoscalingconfig)_ | Autoscaling configures a HorizontalPodAutoscaler for the FeatureStore deployment. +Mutually exclusive with spec.replicas. | + + +#### ScalingStatus + + + +ScalingStatus reports the observed scaling state. + +_Appears in:_ +- [FeatureStoreStatus](#featurestorestatus) + +| Field | Description | +| --- | --- | +| `currentReplicas` _integer_ | CurrentReplicas is the current number of pod replicas. | +| `desiredReplicas` _integer_ | DesiredReplicas is the desired number of pod replicas. | + + #### SecretKeyNames diff --git a/infra/feast-operator/internal/controller/featurestore_controller.go b/infra/feast-operator/internal/controller/featurestore_controller.go index a9591d97c8a..d73b30c0175 100644 --- a/infra/feast-operator/internal/controller/featurestore_controller.go +++ b/infra/feast-operator/internal/controller/featurestore_controller.go @@ -22,6 +22,7 @@ import ( "time" appsv1 "k8s.io/api/apps/v1" + autoscalingv2 "k8s.io/api/autoscaling/v2" batchv1 "k8s.io/api/batch/v1" corev1 "k8s.io/api/core/v1" rbacv1 "k8s.io/api/rbac/v1" @@ -65,6 +66,7 @@ type FeatureStoreReconciler struct { // +kubebuilder:rbac:groups=authentication.k8s.io,resources=tokenreviews,verbs=create // +kubebuilder:rbac:groups=route.openshift.io,resources=routes,verbs=get;list;create;update;watch;delete // +kubebuilder:rbac:groups=batch,resources=cronjobs,verbs=get;list;watch;create;update;patch;delete +// +kubebuilder:rbac:groups=autoscaling,resources=horizontalpodautoscalers,verbs=get;list;watch;create;update;patch;delete // Reconcile is part of the main kubernetes reconciliation loop which aims to // move the current state of the cluster closer to the desired state. @@ -229,6 +231,7 @@ func (r *FeatureStoreReconciler) SetupWithManager(mgr ctrl.Manager) error { Owns(&rbacv1.RoleBinding{}). Owns(&rbacv1.Role{}). Owns(&batchv1.CronJob{}). + Owns(&autoscalingv2.HorizontalPodAutoscaler{}). Watches(&feastdevv1.FeatureStore{}, handler.EnqueueRequestsFromMapFunc(r.mapFeastRefsToFeastRequests)) if services.IsOpenShift() { diff --git a/infra/feast-operator/internal/controller/featurestore_controller_test.go b/infra/feast-operator/internal/controller/featurestore_controller_test.go index bfd4a484cff..a70cd476679 100644 --- a/infra/feast-operator/internal/controller/featurestore_controller_test.go +++ b/infra/feast-operator/internal/controller/featurestore_controller_test.go @@ -263,7 +263,7 @@ var _ = Describe("FeatureStore Controller", func() { Namespace: objMeta.Namespace, }, deploy) Expect(err).NotTo(HaveOccurred()) - Expect(deploy.Spec.Replicas).To(Equal(int32Ptr(3))) + Expect(deploy.Spec.Replicas).To(Equal(int32Ptr(1))) Expect(deploy.Spec.Template.Spec.InitContainers).To(HaveLen(1)) Expect(deploy.Spec.Template.Spec.InitContainers[0].Args[0]).To(ContainSubstring("git -c http.sslVerify=false clone")) Expect(deploy.Spec.Template.Spec.InitContainers[0].Args[0]).To(ContainSubstring("git checkout " + ref)) diff --git a/infra/feast-operator/internal/controller/services/scaling.go b/infra/feast-operator/internal/controller/services/scaling.go new file mode 100644 index 00000000000..b8555555498 --- /dev/null +++ b/infra/feast-operator/internal/controller/services/scaling.go @@ -0,0 +1,204 @@ +/* +Copyright 2026 Feast Community. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package services + +import ( + "encoding/json" + + feastdevv1 "github.com/feast-dev/feast/infra/feast-operator/api/v1" + appsv1 "k8s.io/api/apps/v1" + autoscalingv2 "k8s.io/api/autoscaling/v2" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + hpaac "k8s.io/client-go/applyconfigurations/autoscaling/v2" + metaac "k8s.io/client-go/applyconfigurations/meta/v1" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/log" +) + +const ( + defaultHPACPUUtilization int32 = 80 + defaultHPAMinReplicas int32 = 1 + fieldManager = "feast-operator" +) + +// getDesiredReplicas returns the replica count the operator should set on the +// Deployment. When autoscaling is configured the Deployment replicas field is +// left to the HPA (nil is returned). Otherwise the static replica count from +// spec.replicas is returned. +func (feast *FeastServices) getDesiredReplicas() *int32 { + cr := feast.Handler.FeatureStore + services := cr.Status.Applied.Services + if services != nil && services.Scaling != nil && services.Scaling.Autoscaling != nil { + return nil + } + if cr.Status.Applied.Replicas != nil { + r := *cr.Status.Applied.Replicas + return &r + } + return nil +} + +// createOrDeleteHPA reconciles the HorizontalPodAutoscaler for the FeatureStore +// deployment using Server-Side Apply with typed apply configurations. If +// autoscaling is not configured, any existing HPA is deleted. +func (feast *FeastServices) createOrDeleteHPA() error { + cr := feast.Handler.FeatureStore + + scaling := cr.Status.Applied.Services.Scaling + if scaling == nil || scaling.Autoscaling == nil { + hpa := &autoscalingv2.HorizontalPodAutoscaler{ + ObjectMeta: feast.GetObjectMeta(), + } + hpa.SetGroupVersionKind(autoscalingv2.SchemeGroupVersion.WithKind("HorizontalPodAutoscaler")) + return feast.Handler.DeleteOwnedFeastObj(hpa) + } + + hpaAC := feast.buildHPAApplyConfig() + data, err := json.Marshal(hpaAC) + if err != nil { + return err + } + + hpa := &autoscalingv2.HorizontalPodAutoscaler{ObjectMeta: feast.GetObjectMeta()} + logger := log.FromContext(feast.Handler.Context) + if err := feast.Handler.Client.Patch(feast.Handler.Context, hpa, + client.RawPatch(types.ApplyPatchType, data), + client.FieldOwner(fieldManager), client.ForceOwnership); err != nil { + return err + } + logger.Info("Successfully applied", "HorizontalPodAutoscaler", hpa.Name) + + return nil +} + +// buildHPAApplyConfig constructs the fully desired HPA state as a typed apply +// configuration for Server-Side Apply. +func (feast *FeastServices) buildHPAApplyConfig() *hpaac.HorizontalPodAutoscalerApplyConfiguration { + cr := feast.Handler.FeatureStore + autoscaling := cr.Status.Applied.Services.Scaling.Autoscaling + objMeta := feast.GetObjectMeta() + deploy := feast.initFeastDeploy() + + minReplicas := defaultHPAMinReplicas + if autoscaling.MinReplicas != nil { + minReplicas = *autoscaling.MinReplicas + } + + hpa := hpaac.HorizontalPodAutoscaler(objMeta.Name, objMeta.Namespace). + WithLabels(feast.getLabels()). + WithOwnerReferences( + metaac.OwnerReference(). + WithAPIVersion(feastdevv1.GroupVersion.String()). + WithKind("FeatureStore"). + WithName(cr.Name). + WithUID(cr.UID). + WithController(true). + WithBlockOwnerDeletion(true), + ). + WithSpec(hpaac.HorizontalPodAutoscalerSpec(). + WithScaleTargetRef( + hpaac.CrossVersionObjectReference(). + WithAPIVersion(appsv1.SchemeGroupVersion.String()). + WithKind("Deployment"). + WithName(deploy.Name), + ). + WithMinReplicas(minReplicas). + WithMaxReplicas(autoscaling.MaxReplicas), + ) + + if len(autoscaling.Metrics) > 0 { + hpa.Spec.Metrics = convertMetrics(autoscaling.Metrics) + } else { + hpa.Spec.Metrics = defaultHPAMetrics() + } + + if autoscaling.Behavior != nil { + hpa.Spec.Behavior = convertBehavior(autoscaling.Behavior) + } + + return hpa +} + +func defaultHPAMetrics() []hpaac.MetricSpecApplyConfiguration { + return []hpaac.MetricSpecApplyConfiguration{ + *hpaac.MetricSpec(). + WithType(autoscalingv2.ResourceMetricSourceType). + WithResource( + hpaac.ResourceMetricSource(). + WithName(corev1.ResourceCPU). + WithTarget( + hpaac.MetricTarget(). + WithType(autoscalingv2.UtilizationMetricType). + WithAverageUtilization(defaultHPACPUUtilization), + ), + ), + } +} + +// convertMetrics converts standard API metric specs to their apply configuration +// equivalents via JSON round-trip (the types share identical JSON schemas). +func convertMetrics(metrics []autoscalingv2.MetricSpec) []hpaac.MetricSpecApplyConfiguration { + data, err := json.Marshal(metrics) + if err != nil { + return nil + } + var result []hpaac.MetricSpecApplyConfiguration + if err := json.Unmarshal(data, &result); err != nil { + return nil + } + return result +} + +// convertBehavior converts a standard API behavior spec to its apply configuration +// equivalent via JSON round-trip. +func convertBehavior(behavior *autoscalingv2.HorizontalPodAutoscalerBehavior) *hpaac.HorizontalPodAutoscalerBehaviorApplyConfiguration { + data, err := json.Marshal(behavior) + if err != nil { + return nil + } + result := &hpaac.HorizontalPodAutoscalerBehaviorApplyConfiguration{} + if err := json.Unmarshal(data, result); err != nil { + return nil + } + return result +} + +// updateScalingStatus updates the scaling status fields using the deployment +func (feast *FeastServices) updateScalingStatus(deploy *appsv1.Deployment) { + cr := feast.Handler.FeatureStore + + cr.Status.Replicas = deploy.Status.ReadyReplicas + labels := feast.getLabels() + cr.Status.Selector = metav1.FormatLabelSelector(metav1.SetAsLabelSelector(labels)) + + if !isScalingEnabled(cr) { + cr.Status.ScalingStatus = nil + return + } + + var desired int32 + if deploy.Spec.Replicas != nil { + desired = *deploy.Spec.Replicas + } + + cr.Status.ScalingStatus = &feastdevv1.ScalingStatus{ + CurrentReplicas: deploy.Status.ReadyReplicas, + DesiredReplicas: desired, + } +} diff --git a/infra/feast-operator/internal/controller/services/scaling_test.go b/infra/feast-operator/internal/controller/services/scaling_test.go new file mode 100644 index 00000000000..db803757112 --- /dev/null +++ b/infra/feast-operator/internal/controller/services/scaling_test.go @@ -0,0 +1,725 @@ +/* +Copyright 2026 Feast Community. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package services + +import ( + "context" + + feastdevv1 "github.com/feast-dev/feast/infra/feast-operator/api/v1" + "github.com/feast-dev/feast/infra/feast-operator/internal/controller/handler" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + appsv1 "k8s.io/api/apps/v1" + autoscalingv1 "k8s.io/api/autoscaling/v1" + autoscalingv2 "k8s.io/api/autoscaling/v2" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "sigs.k8s.io/controller-runtime/pkg/client" +) + +var _ = Describe("Horizontal Scaling", func() { + var ( + featureStore *feastdevv1.FeatureStore + feast *FeastServices + typeNamespacedName types.NamespacedName + ctx context.Context + ) + + BeforeEach(func() { + ctx = context.Background() + typeNamespacedName = types.NamespacedName{ + Name: "scaling-test-fs", + Namespace: "default", + } + + featureStore = &feastdevv1.FeatureStore{ + ObjectMeta: metav1.ObjectMeta{ + Name: typeNamespacedName.Name, + Namespace: typeNamespacedName.Namespace, + }, + Spec: feastdevv1.FeatureStoreSpec{ + FeastProject: "scalingproject", + Services: &feastdevv1.FeatureStoreServices{ + OnlineStore: &feastdevv1.OnlineStore{ + Server: &feastdevv1.ServerConfigs{ + ContainerConfigs: feastdevv1.ContainerConfigs{ + DefaultCtrConfigs: feastdevv1.DefaultCtrConfigs{ + Image: ptr("test-image"), + }, + }, + }, + Persistence: &feastdevv1.OnlineStorePersistence{ + DBPersistence: &feastdevv1.OnlineStoreDBStorePersistence{ + Type: "redis", + SecretRef: corev1.LocalObjectReference{ + Name: "redis-secret", + }, + }, + }, + }, + Registry: &feastdevv1.Registry{ + Local: &feastdevv1.LocalRegistryConfig{ + Server: &feastdevv1.RegistryServerConfigs{ + ServerConfigs: feastdevv1.ServerConfigs{ + ContainerConfigs: feastdevv1.ContainerConfigs{ + DefaultCtrConfigs: feastdevv1.DefaultCtrConfigs{ + Image: ptr("test-image"), + }, + }, + }, + GRPC: ptr(true), + }, + Persistence: &feastdevv1.RegistryPersistence{ + DBPersistence: &feastdevv1.RegistryDBStorePersistence{ + Type: "sql", + SecretRef: corev1.LocalObjectReference{ + Name: "registry-secret", + }, + }, + }, + }, + }, + }, + }, + } + + Expect(k8sClient.Create(ctx, featureStore)).To(Succeed()) + applySpecToStatus(featureStore) + + feast = &FeastServices{ + Handler: handler.FeastHandler{ + Client: k8sClient, + Context: ctx, + Scheme: k8sClient.Scheme(), + FeatureStore: featureStore, + }, + } + }) + + AfterEach(func() { + Expect(k8sClient.Delete(ctx, featureStore)).To(Succeed()) + }) + + Describe("isScalingEnabled", func() { + It("should return false when no scaling config is present", func() { + Expect(isScalingEnabled(featureStore)).To(BeFalse()) + }) + + It("should return false when replicas=1", func() { + featureStore.Status.Applied.Replicas = ptr(int32(1)) + Expect(isScalingEnabled(featureStore)).To(BeFalse()) + }) + + It("should return true when replicas > 1", func() { + featureStore.Status.Applied.Replicas = ptr(int32(3)) + Expect(isScalingEnabled(featureStore)).To(BeTrue()) + }) + + It("should return true when autoscaling is configured", func() { + featureStore.Status.Applied.Services.Scaling = &feastdevv1.ScalingConfig{ + Autoscaling: &feastdevv1.AutoscalingConfig{ + MaxReplicas: 5, + }, + } + Expect(isScalingEnabled(featureStore)).To(BeTrue()) + }) + }) + + Describe("CEL admission validation rejects invalid scaling configurations", func() { + dbOnlineStore := &feastdevv1.OnlineStore{ + Persistence: &feastdevv1.OnlineStorePersistence{ + DBPersistence: &feastdevv1.OnlineStoreDBStorePersistence{ + Type: "redis", + SecretRef: corev1.LocalObjectReference{Name: "redis-secret"}, + }, + }, + } + + dbRegistry := &feastdevv1.Registry{ + Local: &feastdevv1.LocalRegistryConfig{ + Persistence: &feastdevv1.RegistryPersistence{ + DBPersistence: &feastdevv1.RegistryDBStorePersistence{ + Type: "sql", + SecretRef: corev1.LocalObjectReference{Name: "registry-secret"}, + }, + }, + }, + } + + It("should accept scaling with full DB persistence", func() { + fs := &feastdevv1.FeatureStore{ + ObjectMeta: metav1.ObjectMeta{Name: "cel-valid-db", Namespace: "default"}, + Spec: feastdevv1.FeatureStoreSpec{ + FeastProject: "celtest", + Replicas: ptr(int32(3)), + Services: &feastdevv1.FeatureStoreServices{ + OnlineStore: dbOnlineStore, + Registry: dbRegistry, + }, + }, + } + Expect(k8sClient.Create(ctx, fs)).To(Succeed()) + Expect(k8sClient.Delete(ctx, fs)).To(Succeed()) + }) + + It("should reject scaling when online store is missing (implicit file default)", func() { + fs := &feastdevv1.FeatureStore{ + ObjectMeta: metav1.ObjectMeta{Name: "cel-no-online", Namespace: "default"}, + Spec: feastdevv1.FeatureStoreSpec{ + FeastProject: "celtest", + Replicas: ptr(int32(3)), + Services: &feastdevv1.FeatureStoreServices{ + Registry: dbRegistry, + }, + }, + } + err := k8sClient.Create(ctx, fs) + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(ContainSubstring("online store")) + }) + + It("should reject scaling when online store uses file persistence", func() { + fs := &feastdevv1.FeatureStore{ + ObjectMeta: metav1.ObjectMeta{Name: "cel-file-online", Namespace: "default"}, + Spec: feastdevv1.FeatureStoreSpec{ + FeastProject: "celtest", + Replicas: ptr(int32(3)), + Services: &feastdevv1.FeatureStoreServices{ + OnlineStore: &feastdevv1.OnlineStore{ + Persistence: &feastdevv1.OnlineStorePersistence{ + FilePersistence: &feastdevv1.OnlineStoreFilePersistence{ + Path: "/data/online.db", + }, + }, + }, + Registry: dbRegistry, + }, + }, + } + err := k8sClient.Create(ctx, fs) + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(ContainSubstring("online store")) + }) + + It("should reject scaling when offline store uses file persistence", func() { + fs := &feastdevv1.FeatureStore{ + ObjectMeta: metav1.ObjectMeta{Name: "cel-file-offline", Namespace: "default"}, + Spec: feastdevv1.FeatureStoreSpec{ + FeastProject: "celtest", + Replicas: ptr(int32(3)), + Services: &feastdevv1.FeatureStoreServices{ + OnlineStore: dbOnlineStore, + Registry: dbRegistry, + OfflineStore: &feastdevv1.OfflineStore{ + Persistence: &feastdevv1.OfflineStorePersistence{ + FilePersistence: &feastdevv1.OfflineStoreFilePersistence{ + Type: "duckdb", + }, + }, + }, + }, + }, + } + err := k8sClient.Create(ctx, fs) + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(ContainSubstring("offline store")) + }) + + It("should reject scaling when no registry is configured (implicit file default)", func() { + fs := &feastdevv1.FeatureStore{ + ObjectMeta: metav1.ObjectMeta{Name: "cel-no-registry", Namespace: "default"}, + Spec: feastdevv1.FeatureStoreSpec{ + FeastProject: "celtest", + Replicas: ptr(int32(3)), + Services: &feastdevv1.FeatureStoreServices{ + OnlineStore: dbOnlineStore, + }, + }, + } + err := k8sClient.Create(ctx, fs) + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(ContainSubstring("registry")) + }) + + It("should reject scaling when registry uses file persistence", func() { + fs := &feastdevv1.FeatureStore{ + ObjectMeta: metav1.ObjectMeta{Name: "cel-file-registry", Namespace: "default"}, + Spec: feastdevv1.FeatureStoreSpec{ + FeastProject: "celtest", + Replicas: ptr(int32(3)), + Services: &feastdevv1.FeatureStoreServices{ + OnlineStore: dbOnlineStore, + Registry: &feastdevv1.Registry{ + Local: &feastdevv1.LocalRegistryConfig{ + Persistence: &feastdevv1.RegistryPersistence{ + FilePersistence: &feastdevv1.RegistryFilePersistence{ + Path: "/data/registry.db", + }, + }, + }, + }, + }, + }, + } + err := k8sClient.Create(ctx, fs) + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(ContainSubstring("registry")) + }) + + It("should accept scaling with S3-backed registry", func() { + fs := &feastdevv1.FeatureStore{ + ObjectMeta: metav1.ObjectMeta{Name: "cel-s3-registry", Namespace: "default"}, + Spec: feastdevv1.FeatureStoreSpec{ + FeastProject: "celtest", + Replicas: ptr(int32(3)), + Services: &feastdevv1.FeatureStoreServices{ + OnlineStore: dbOnlineStore, + Registry: &feastdevv1.Registry{ + Local: &feastdevv1.LocalRegistryConfig{ + Persistence: &feastdevv1.RegistryPersistence{ + FilePersistence: &feastdevv1.RegistryFilePersistence{ + Path: "s3://my-bucket/registry.db", + }, + }, + }, + }, + }, + }, + } + Expect(k8sClient.Create(ctx, fs)).To(Succeed()) + Expect(k8sClient.Delete(ctx, fs)).To(Succeed()) + }) + + It("should accept scaling with GS-backed registry", func() { + fs := &feastdevv1.FeatureStore{ + ObjectMeta: metav1.ObjectMeta{Name: "cel-gs-registry", Namespace: "default"}, + Spec: feastdevv1.FeatureStoreSpec{ + FeastProject: "celtest", + Replicas: ptr(int32(3)), + Services: &feastdevv1.FeatureStoreServices{ + OnlineStore: dbOnlineStore, + Registry: &feastdevv1.Registry{ + Local: &feastdevv1.LocalRegistryConfig{ + Persistence: &feastdevv1.RegistryPersistence{ + FilePersistence: &feastdevv1.RegistryFilePersistence{ + Path: "gs://my-bucket/registry.db", + }, + }, + }, + }, + }, + }, + } + Expect(k8sClient.Create(ctx, fs)).To(Succeed()) + Expect(k8sClient.Delete(ctx, fs)).To(Succeed()) + }) + + It("should accept scaling with remote registry", func() { + fs := &feastdevv1.FeatureStore{ + ObjectMeta: metav1.ObjectMeta{Name: "cel-remote-reg", Namespace: "default"}, + Spec: feastdevv1.FeatureStoreSpec{ + FeastProject: "celtest", + Replicas: ptr(int32(3)), + Services: &feastdevv1.FeatureStoreServices{ + OnlineStore: dbOnlineStore, + Registry: &feastdevv1.Registry{ + Remote: &feastdevv1.RemoteRegistryConfig{ + Hostname: ptr("registry.example.com:80"), + }, + }, + }, + }, + } + Expect(k8sClient.Create(ctx, fs)).To(Succeed()) + Expect(k8sClient.Delete(ctx, fs)).To(Succeed()) + }) + + It("should accept file persistence when replicas is 1", func() { + fs := &feastdevv1.FeatureStore{ + ObjectMeta: metav1.ObjectMeta{Name: "cel-rep1-file", Namespace: "default"}, + Spec: feastdevv1.FeatureStoreSpec{ + FeastProject: "celtest", + Replicas: ptr(int32(1)), + }, + } + Expect(k8sClient.Create(ctx, fs)).To(Succeed()) + Expect(k8sClient.Delete(ctx, fs)).To(Succeed()) + }) + + It("should accept file persistence when no scaling is configured", func() { + fs := &feastdevv1.FeatureStore{ + ObjectMeta: metav1.ObjectMeta{Name: "cel-no-scaling", Namespace: "default"}, + Spec: feastdevv1.FeatureStoreSpec{ + FeastProject: "celtest", + }, + } + Expect(k8sClient.Create(ctx, fs)).To(Succeed()) + Expect(k8sClient.Delete(ctx, fs)).To(Succeed()) + }) + + It("should reject autoscaling without DB online store", func() { + fs := &feastdevv1.FeatureStore{ + ObjectMeta: metav1.ObjectMeta{Name: "cel-hpa-no-db", Namespace: "default"}, + Spec: feastdevv1.FeatureStoreSpec{ + FeastProject: "celtest", + Services: &feastdevv1.FeatureStoreServices{ + Scaling: &feastdevv1.ScalingConfig{ + Autoscaling: &feastdevv1.AutoscalingConfig{MaxReplicas: 5}, + }, + Registry: dbRegistry, + }, + }, + } + err := k8sClient.Create(ctx, fs) + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(ContainSubstring("online store")) + }) + + It("should reject scaling when online store has no persistence configured", func() { + fs := &feastdevv1.FeatureStore{ + ObjectMeta: metav1.ObjectMeta{Name: "cel-online-nop", Namespace: "default"}, + Spec: feastdevv1.FeatureStoreSpec{ + FeastProject: "celtest", + Replicas: ptr(int32(3)), + Services: &feastdevv1.FeatureStoreServices{ + OnlineStore: &feastdevv1.OnlineStore{}, + Registry: dbRegistry, + }, + }, + } + err := k8sClient.Create(ctx, fs) + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(ContainSubstring("online store")) + }) + + It("should reject replicas and autoscaling set simultaneously", func() { + fs := &feastdevv1.FeatureStore{ + ObjectMeta: metav1.ObjectMeta{Name: "cel-mutual-excl", Namespace: "default"}, + Spec: feastdevv1.FeatureStoreSpec{ + FeastProject: "celtest", + Replicas: ptr(int32(3)), + Services: &feastdevv1.FeatureStoreServices{ + Scaling: &feastdevv1.ScalingConfig{ + Autoscaling: &feastdevv1.AutoscalingConfig{MaxReplicas: 5}, + }, + OnlineStore: dbOnlineStore, + Registry: dbRegistry, + }, + }, + } + err := k8sClient.Create(ctx, fs) + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(ContainSubstring("mutually exclusive")) + }) + }) + + Describe("getDesiredReplicas", func() { + It("should return 1 when no explicit replicas are configured (default)", func() { + replicas := feast.getDesiredReplicas() + Expect(replicas).NotTo(BeNil()) + Expect(*replicas).To(Equal(int32(1))) + }) + + It("should return static replicas when configured", func() { + featureStore.Status.Applied.Replicas = ptr(int32(3)) + replicas := feast.getDesiredReplicas() + Expect(replicas).NotTo(BeNil()) + Expect(*replicas).To(Equal(int32(3))) + }) + + It("should return nil when autoscaling is configured (HPA manages replicas)", func() { + featureStore.Status.Applied.Services.Scaling = &feastdevv1.ScalingConfig{ + Autoscaling: &feastdevv1.AutoscalingConfig{ + MaxReplicas: 5, + }, + } + Expect(feast.getDesiredReplicas()).To(BeNil()) + }) + }) + + Describe("Deployment Strategy", func() { + It("should default to Recreate when no scaling is configured", func() { + Expect(feast.ApplyDefaults()).To(Succeed()) + strategy := feast.getDeploymentStrategy() + Expect(strategy.Type).To(Equal(appsv1.RecreateDeploymentStrategyType)) + }) + + It("should default to RollingUpdate when scaling is enabled via replicas", func() { + featureStore.Status.Applied.Replicas = ptr(int32(3)) + strategy := feast.getDeploymentStrategy() + Expect(strategy.Type).To(Equal(appsv1.RollingUpdateDeploymentStrategyType)) + }) + + It("should respect user-defined strategy even with scaling", func() { + featureStore.Status.Applied.Replicas = ptr(int32(3)) + featureStore.Status.Applied.Services.DeploymentStrategy = &appsv1.DeploymentStrategy{ + Type: appsv1.RecreateDeploymentStrategyType, + } + strategy := feast.getDeploymentStrategy() + Expect(strategy.Type).To(Equal(appsv1.RecreateDeploymentStrategyType)) + }) + }) + + Describe("setDeployment with scaling", func() { + setFilePersistence := func() { + featureStore.Status.Applied.Services.OnlineStore = &feastdevv1.OnlineStore{ + Server: &feastdevv1.ServerConfigs{ + ContainerConfigs: feastdevv1.ContainerConfigs{ + DefaultCtrConfigs: feastdevv1.DefaultCtrConfigs{ + Image: ptr("test-image"), + }, + }, + }, + Persistence: &feastdevv1.OnlineStorePersistence{ + FilePersistence: &feastdevv1.OnlineStoreFilePersistence{ + Path: "/feast-data/online.db", + }, + }, + } + featureStore.Status.Applied.Services.Registry = &feastdevv1.Registry{ + Local: &feastdevv1.LocalRegistryConfig{ + Server: &feastdevv1.RegistryServerConfigs{ + ServerConfigs: feastdevv1.ServerConfigs{ + ContainerConfigs: feastdevv1.ContainerConfigs{ + DefaultCtrConfigs: feastdevv1.DefaultCtrConfigs{ + Image: ptr("test-image"), + }, + }, + }, + GRPC: ptr(true), + }, + Persistence: &feastdevv1.RegistryPersistence{ + FilePersistence: &feastdevv1.RegistryFilePersistence{ + Path: "/feast-data/registry.db", + }, + }, + }, + } + } + + It("should set static replicas on the deployment", func() { + setFilePersistence() + featureStore.Status.Applied.Replicas = ptr(int32(3)) + + deployment := feast.initFeastDeploy() + Expect(feast.setDeployment(deployment)).To(Succeed()) + Expect(deployment.Spec.Replicas).NotTo(BeNil()) + Expect(*deployment.Spec.Replicas).To(Equal(int32(3))) + }) + + It("should preserve existing replicas when autoscaling is configured", func() { + setFilePersistence() + featureStore.Status.Applied.Services.Scaling = &feastdevv1.ScalingConfig{ + Autoscaling: &feastdevv1.AutoscalingConfig{ + MaxReplicas: 5, + }, + } + + deployment := feast.initFeastDeploy() + existing := int32(4) + deployment.Spec.Replicas = &existing + Expect(feast.setDeployment(deployment)).To(Succeed()) + Expect(deployment.Spec.Replicas).NotTo(BeNil()) + Expect(*deployment.Spec.Replicas).To(Equal(int32(4))) + }) + + It("should set default replicas=1 when no explicit scaling is configured", func() { + setFilePersistence() + Expect(k8sClient.Status().Update(ctx, featureStore)).To(Succeed()) + feast.refreshFeatureStore(ctx, typeNamespacedName) + + deployment := feast.initFeastDeploy() + Expect(feast.setDeployment(deployment)).To(Succeed()) + Expect(deployment.Spec.Replicas).NotTo(BeNil()) + Expect(*deployment.Spec.Replicas).To(Equal(int32(1))) + }) + }) + + Describe("HPA Configuration", func() { + It("should build an HPA apply config with default CPU metrics", func() { + featureStore.Status.Applied.Services.Scaling = &feastdevv1.ScalingConfig{ + Autoscaling: &feastdevv1.AutoscalingConfig{ + MaxReplicas: 10, + }, + } + + hpa := feast.buildHPAApplyConfig() + Expect(*hpa.Spec.MaxReplicas).To(Equal(int32(10))) + Expect(*hpa.Spec.MinReplicas).To(Equal(int32(1))) + Expect(hpa.Spec.Metrics).To(HaveLen(1)) + Expect(*hpa.Spec.Metrics[0].Resource.Name).To(Equal(corev1.ResourceCPU)) + }) + + It("should build an HPA apply config with custom min replicas", func() { + featureStore.Status.Applied.Services.Scaling = &feastdevv1.ScalingConfig{ + Autoscaling: &feastdevv1.AutoscalingConfig{ + MinReplicas: ptr(int32(2)), + MaxReplicas: 10, + }, + } + + hpa := feast.buildHPAApplyConfig() + Expect(*hpa.Spec.MinReplicas).To(Equal(int32(2))) + Expect(*hpa.Spec.MaxReplicas).To(Equal(int32(10))) + }) + + It("should set correct scale target reference", func() { + featureStore.Status.Applied.Services.Scaling = &feastdevv1.ScalingConfig{ + Autoscaling: &feastdevv1.AutoscalingConfig{ + MaxReplicas: 5, + }, + } + + hpa := feast.buildHPAApplyConfig() + Expect(*hpa.Spec.ScaleTargetRef.APIVersion).To(Equal("apps/v1")) + Expect(*hpa.Spec.ScaleTargetRef.Kind).To(Equal("Deployment")) + Expect(*hpa.Spec.ScaleTargetRef.Name).To(Equal(GetFeastName(featureStore))) + }) + + It("should set TypeMeta and owner reference for SSA", func() { + featureStore.Status.Applied.Services.Scaling = &feastdevv1.ScalingConfig{ + Autoscaling: &feastdevv1.AutoscalingConfig{ + MaxReplicas: 5, + }, + } + + hpa := feast.buildHPAApplyConfig() + Expect(*hpa.Kind).To(Equal("HorizontalPodAutoscaler")) + Expect(*hpa.APIVersion).To(Equal("autoscaling/v2")) + Expect(hpa.OwnerReferences).To(HaveLen(1)) + Expect(*hpa.OwnerReferences[0].Name).To(Equal(featureStore.Name)) + Expect(*hpa.OwnerReferences[0].Controller).To(BeTrue()) + }) + + It("should convert custom metrics via JSON round-trip", func() { + customMetrics := []autoscalingv2.MetricSpec{ + { + Type: autoscalingv2.ResourceMetricSourceType, + Resource: &autoscalingv2.ResourceMetricSource{ + Name: corev1.ResourceMemory, + Target: autoscalingv2.MetricTarget{ + Type: autoscalingv2.UtilizationMetricType, + AverageUtilization: ptr(int32(75)), + }, + }, + }, + } + featureStore.Status.Applied.Services.Scaling = &feastdevv1.ScalingConfig{ + Autoscaling: &feastdevv1.AutoscalingConfig{ + MaxReplicas: 10, + Metrics: customMetrics, + }, + } + + hpa := feast.buildHPAApplyConfig() + Expect(hpa.Spec.Metrics).To(HaveLen(1)) + Expect(*hpa.Spec.Metrics[0].Resource.Name).To(Equal(corev1.ResourceMemory)) + Expect(*hpa.Spec.Metrics[0].Resource.Target.AverageUtilization).To(Equal(int32(75))) + }) + }) + + Describe("Scale sub-resource", func() { + newDBFeatureStore := func(name string) *feastdevv1.FeatureStore { + return &feastdevv1.FeatureStore{ + ObjectMeta: metav1.ObjectMeta{ + Name: name, + Namespace: "default", + }, + Spec: feastdevv1.FeatureStoreSpec{ + FeastProject: "scaletest", + Services: &feastdevv1.FeatureStoreServices{ + OnlineStore: &feastdevv1.OnlineStore{ + Persistence: &feastdevv1.OnlineStorePersistence{ + DBPersistence: &feastdevv1.OnlineStoreDBStorePersistence{ + Type: "redis", + SecretRef: corev1.LocalObjectReference{Name: "redis-secret"}, + }, + }, + }, + Registry: &feastdevv1.Registry{ + Local: &feastdevv1.LocalRegistryConfig{ + Persistence: &feastdevv1.RegistryPersistence{ + DBPersistence: &feastdevv1.RegistryDBStorePersistence{ + Type: "sql", + SecretRef: corev1.LocalObjectReference{Name: "registry-secret"}, + }, + }, + }, + }, + }, + }, + } + } + + It("should allow scaling up via the scale sub-resource with DB persistence", func() { + fs := newDBFeatureStore("scale-sub-valid") + Expect(k8sClient.Create(ctx, fs)).To(Succeed()) + defer func() { Expect(k8sClient.Delete(ctx, fs)).To(Succeed()) }() + + scale := &autoscalingv1.Scale{} + Expect(k8sClient.SubResource("scale").Get(ctx, fs, scale)).To(Succeed()) + Expect(scale.Spec.Replicas).To(Equal(int32(1))) + + scale.Spec.Replicas = 3 + Expect(k8sClient.SubResource("scale").Update(ctx, fs, client.WithSubResourceBody(scale))).To(Succeed()) + + updated := &feastdevv1.FeatureStore{} + Expect(k8sClient.Get(ctx, types.NamespacedName{Name: fs.Name, Namespace: fs.Namespace}, updated)).To(Succeed()) + Expect(updated.Spec.Replicas).NotTo(BeNil()) + Expect(*updated.Spec.Replicas).To(Equal(int32(3))) + }) + + It("should reject scaling up via the scale sub-resource without DB persistence", func() { + fs := &feastdevv1.FeatureStore{ + ObjectMeta: metav1.ObjectMeta{ + Name: "scale-sub-reject", + Namespace: "default", + }, + Spec: feastdevv1.FeatureStoreSpec{ + FeastProject: "scaletest", + }, + } + Expect(k8sClient.Create(ctx, fs)).To(Succeed()) + defer func() { Expect(k8sClient.Delete(ctx, fs)).To(Succeed()) }() + + scale := &autoscalingv1.Scale{} + Expect(k8sClient.SubResource("scale").Get(ctx, fs, scale)).To(Succeed()) + + scale.Spec.Replicas = 3 + err := k8sClient.SubResource("scale").Update(ctx, fs, client.WithSubResourceBody(scale)) + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(ContainSubstring("online store")) + }) + + It("should read the status replicas from the scale sub-resource", func() { + fs := newDBFeatureStore("scale-sub-status") + fs.Spec.Replicas = ptr(int32(2)) + Expect(k8sClient.Create(ctx, fs)).To(Succeed()) + defer func() { Expect(k8sClient.Delete(ctx, fs)).To(Succeed()) }() + + fs.Status.Replicas = 2 + fs.Status.Applied.FeastProject = fs.Spec.FeastProject + Expect(k8sClient.Status().Update(ctx, fs)).To(Succeed()) + + scale := &autoscalingv1.Scale{} + Expect(k8sClient.SubResource("scale").Get(ctx, fs, scale)).To(Succeed()) + Expect(scale.Status.Replicas).To(Equal(int32(2))) + Expect(scale.Spec.Replicas).To(Equal(int32(2))) + }) + }) +}) diff --git a/infra/feast-operator/internal/controller/services/services.go b/infra/feast-operator/internal/controller/services/services.go index 6771e9498af..a76f21d18c8 100644 --- a/infra/feast-operator/internal/controller/services/services.go +++ b/infra/feast-operator/internal/controller/services/services.go @@ -71,14 +71,42 @@ func (feast *FeastServices) Deploy() error { _ = feast.Handler.DeleteOwnedFeastObj(feast.initCaConfigMap()) } + if err := feast.reconcileServices(); err != nil { + return err + } + + if err := feast.createServiceAccount(); err != nil { + return err + } + if err := feast.createDeployment(); err != nil { + return err + } + if err := feast.createOrDeleteHPA(); err != nil { + return err + } + if err := feast.deployClient(); err != nil { + return err + } + if err := feast.deployNamespaceRegistry(); err != nil { + return err + } + if err := feast.deployCronJob(); err != nil { + return err + } + + return nil +} + +// reconcileServices validates persistence and deploys or removes each feast +// service type based on the applied spec. +func (feast *FeastServices) reconcileServices() error { services := feast.Handler.FeatureStore.Status.Applied.Services + if feast.isOfflineStore() { - err := feast.validateOfflineStorePersistence(services.OfflineStore.Persistence) - if err != nil { + if err := feast.validateOfflineStorePersistence(services.OfflineStore.Persistence); err != nil { return err } - - if err = feast.deployFeastServiceByType(OfflineFeastType); err != nil { + if err := feast.deployFeastServiceByType(OfflineFeastType); err != nil { return err } } else { @@ -88,12 +116,10 @@ func (feast *FeastServices) Deploy() error { } if feast.isOnlineStore() { - err := feast.validateOnlineStorePersistence(services.OnlineStore.Persistence) - if err != nil { + if err := feast.validateOnlineStorePersistence(services.OnlineStore.Persistence); err != nil { return err } - - if err = feast.deployFeastServiceByType(OnlineFeastType); err != nil { + if err := feast.deployFeastServiceByType(OnlineFeastType); err != nil { return err } } else { @@ -103,12 +129,10 @@ func (feast *FeastServices) Deploy() error { } if feast.isLocalRegistry() { - err := feast.validateRegistryPersistence(services.Registry.Local.Persistence) - if err != nil { + if err := feast.validateRegistryPersistence(services.Registry.Local.Persistence); err != nil { return err } - - if err = feast.deployFeastServiceByType(RegistryFeastType); err != nil { + if err := feast.deployFeastServiceByType(RegistryFeastType); err != nil { return err } } else { @@ -116,11 +140,12 @@ func (feast *FeastServices) Deploy() error { return err } } + if feast.isUiServer() { - if err = feast.deployFeastServiceByType(UIFeastType); err != nil { + if err := feast.deployFeastServiceByType(UIFeastType); err != nil { return err } - if err = feast.createRoute(UIFeastType); err != nil { + if err := feast.createRoute(UIFeastType); err != nil { return err } } else { @@ -132,22 +157,6 @@ func (feast *FeastServices) Deploy() error { } } - if err := feast.createServiceAccount(); err != nil { - return err - } - if err := feast.createDeployment(); err != nil { - return err - } - if err := feast.deployClient(); err != nil { - return err - } - if err := feast.deployNamespaceRegistry(); err != nil { - return err - } - if err := feast.deployCronJob(); err != nil { - return err - } - return nil } @@ -338,6 +347,8 @@ func (feast *FeastServices) createDeployment() error { logger.Info("Successfully reconciled", "Deployment", deploy.Name, "operation", op) } + feast.updateScalingStatus(deploy) + return nil } @@ -381,7 +392,14 @@ func (feast *FeastServices) createPVC(pvcCreate *feastdevv1.PvcCreate, feastType func (feast *FeastServices) setDeployment(deploy *appsv1.Deployment) error { cr := feast.Handler.FeatureStore + + // Determine replica count: + // - spec.replicas is set on the Deployment (defaults to 1) + // - When HPA is configured, replicas is left unset so the HPA controller manages it replicas := deploy.Spec.Replicas + if desired := feast.getDesiredReplicas(); desired != nil { + replicas = desired + } deploy.Labels = feast.getLabels() deploy.Spec = appsv1.DeploymentSpec{ @@ -635,6 +653,11 @@ func (feast *FeastServices) getDeploymentStrategy() appsv1.DeploymentStrategy { if feast.Handler.FeatureStore.Status.Applied.Services.DeploymentStrategy != nil { return *feast.Handler.FeatureStore.Status.Applied.Services.DeploymentStrategy } + if isScalingEnabled(feast.Handler.FeatureStore) { + return appsv1.DeploymentStrategy{ + Type: appsv1.RollingUpdateDeploymentStrategyType, + } + } return appsv1.DeploymentStrategy{ Type: appsv1.RecreateDeploymentStrategyType, } diff --git a/infra/feast-operator/internal/controller/services/util.go b/infra/feast-operator/internal/controller/services/util.go index 33d750251e9..9ce1ecd749a 100644 --- a/infra/feast-operator/internal/controller/services/util.go +++ b/infra/feast-operator/internal/controller/services/util.go @@ -493,6 +493,16 @@ func getVolumeMountByType(feastType FeastServiceType, featureStore *feastdevv1.F return nil } +// isScalingEnabled returns true when the user has configured horizontal scaling +// with either static replicas > 1 or HPA autoscaling. +func isScalingEnabled(featureStore *feastdevv1.FeatureStore) bool { + if featureStore.Status.Applied.Replicas != nil && *featureStore.Status.Applied.Replicas > 1 { + return true + } + services := featureStore.Status.Applied.Services + return services != nil && services.Scaling != nil && services.Scaling.Autoscaling != nil +} + func boolPtr(value bool) *bool { return &value } diff --git a/infra/website/docs/blog/scaling-feast-feature-server.md b/infra/website/docs/blog/scaling-feast-feature-server.md new file mode 100644 index 00000000000..4406d280c81 --- /dev/null +++ b/infra/website/docs/blog/scaling-feast-feature-server.md @@ -0,0 +1,252 @@ +--- +title: Scaling the Feast Feature Server on Kubernetes +description: The Feast Operator now supports horizontal scaling with static replicas, HPA autoscaling, and external autoscalers like KEDA — enabling production-grade, high-availability feature serving. +date: 2026-02-21 +authors: ["Nikhil Kathole"] +--- + +# Scaling the Feast Feature Server on Kubernetes + +As ML systems move from experimentation to production, the feature server often becomes a critical bottleneck. A single-replica deployment might handle development traffic, but production workloads — real-time inference, batch scoring, multiple consuming services — demand the ability to scale horizontally. + +We're excited to announce that the Feast Operator now supports **horizontal scaling** for the FeatureStore deployment, giving teams the tools to run Feast at production scale on Kubernetes. + +# The Problem: Single-Replica Limitations + +By default, the Feast Operator deploys a single-replica Deployment. This works well for getting started, but presents challenges as traffic grows: + +- **Single point of failure** — one pod crash means downtime for all feature consumers +- **Throughput ceiling** — a single pod can only handle so many concurrent requests +- **No elasticity** — traffic spikes (model retraining, batch inference) can overwhelm the server +- **Rolling updates cause downtime** — the default `Recreate` strategy tears down the old pod before starting a new one + +Teams have been manually patching Deployments or creating external HPAs, but this bypasses the operator's reconciliation loop and can lead to configuration drift. + +# The Solution: Native Scaling Support + +The Feast Operator now supports three scaling modes. The FeatureStore CRD implements the Kubernetes **scale sub-resource**, which means you can also scale with `kubectl scale featurestore/my-feast --replicas=3`. + +## 1. Static Replicas + +The simplest approach — set a fixed number of replicas via `spec.replicas`: + +```yaml +apiVersion: feast.dev/v1 +kind: FeatureStore +metadata: + name: production-feast +spec: + feastProject: my_project + replicas: 3 + services: + onlineStore: + persistence: + store: + type: postgres + secretRef: + name: feast-data-stores + registry: + local: + persistence: + store: + type: sql + secretRef: + name: feast-data-stores +``` + +This gives you high availability and load distribution with a predictable resource footprint. The operator automatically switches the Deployment strategy to `RollingUpdate`, ensuring zero-downtime deployments. + +## 2. HPA Autoscaling + +For workloads with variable traffic patterns, the operator can create and manage a `HorizontalPodAutoscaler` directly. HPA autoscaling is configured under `services.scaling.autoscaling` and is mutually exclusive with `spec.replicas > 1`: + +```yaml +apiVersion: feast.dev/v1 +kind: FeatureStore +metadata: + name: autoscaled-feast +spec: + feastProject: my_project + services: + scaling: + autoscaling: + minReplicas: 2 + maxReplicas: 10 + metrics: + - type: Resource + resource: + name: cpu + target: + type: Utilization + averageUtilization: 70 + onlineStore: + persistence: + store: + type: postgres + secretRef: + name: feast-data-stores + server: + resources: + requests: + cpu: 200m + memory: 256Mi + limits: + cpu: "1" + memory: 1Gi + registry: + local: + persistence: + store: + type: sql + secretRef: + name: feast-data-stores +``` + +The operator creates the HPA as an owned resource — it's automatically cleaned up if you remove the autoscaling configuration or delete the FeatureStore CR. If no custom metrics are specified, the operator defaults to **80% CPU utilization**. + +## 3. External Autoscalers (KEDA, Custom HPAs) + +For teams using [KEDA](https://keda.sh) or other external autoscalers, KEDA should target the FeatureStore's scale sub-resource directly (since it implements the Kubernetes scale API). This is the recommended approach because the operator manages the Deployment's replica count from `spec.replicas` — targeting the Deployment directly would conflict with the operator's reconciliation. + +When using KEDA, do **not** set `spec.replicas > 1` or `services.scaling.autoscaling` — KEDA manages the replica count through the scale sub-resource. Configure the FeatureStore with DB-backed persistence, then create a KEDA `ScaledObject` targeting the FeatureStore resource: + +```yaml +apiVersion: feast.dev/v1 +kind: FeatureStore +metadata: + name: keda-feast +spec: + feastProject: my_project + services: + onlineStore: + persistence: + store: + type: postgres + secretRef: + name: feast-data-stores + registry: + local: + persistence: + store: + type: sql + secretRef: + name: feast-data-stores +--- +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: feast-scaledobject +spec: + scaleTargetRef: + apiVersion: feast.dev/v1 + kind: FeatureStore + name: keda-feast + minReplicaCount: 1 + maxReplicaCount: 10 + triggers: + - type: prometheus + metadata: + serverAddress: http://prometheus.monitoring.svc:9090 + metricName: http_requests_total + query: sum(rate(http_requests_total{service="feast"}[2m])) + threshold: "100" +``` + +When KEDA scales up `spec.replicas` via the scale sub-resource, the CRD's CEL validation rules automatically ensure DB-backed persistence is configured. The operator also automatically switches the deployment strategy to `RollingUpdate` when `replicas > 1`. This gives you the full power of KEDA's 50+ event-driven triggers with built-in safety checks. + +# Safety First: Persistence Validation + +Not all persistence backends are safe for multi-replica deployments. File-based stores like SQLite, DuckDB, and local `registry.db` use single-writer file locks that don't work across pods. + +The operator enforces this at admission time via CEL validation rules on the CRD — if you try to create or update a FeatureStore with scaling and file-based persistence, the API server rejects the request immediately: + +``` +Scaling requires DB-backed persistence for the online store. +Configure services.onlineStore.persistence.store when using replicas > 1 or autoscaling. +``` + +This validation applies to all enabled services (online store, offline store, and registry) and is enforced for both direct CR updates and `kubectl scale` commands via the scale sub-resource. Object-store-backed registry paths (`s3://` and `gs://`) are treated as safe since they support concurrent readers. + +| Persistence Type | Compatible with Scaling? | +|---|---| +| PostgreSQL / MySQL | Yes | +| Redis | Yes | +| Cassandra | Yes | +| SQL-based Registry | Yes | +| S3/GCS Registry | Yes | +| SQLite | No | +| DuckDB | No | +| Local `registry.db` | No | + +# How It Works Under the Hood + +The implementation adds three key behaviors to the operator's reconciliation loop: + +**1. Replica management** — The operator sets the Deployment's replica count from `spec.replicas` (which defaults to 1). When HPA is configured, the operator leaves the `replicas` field unset so the HPA controller can manage it. External autoscalers like KEDA can update the replica count through the FeatureStore's scale sub-resource, which updates `spec.replicas` and triggers the operator to reconcile. + +**2. Deployment strategy** — The operator automatically switches from `Recreate` (the default for single-replica) to `RollingUpdate` when scaling is enabled. This prevents the "kill-all-pods-then-start-new-ones" behavior that would cause downtime during scaling events. Users can always override this with an explicit `deploymentStrategy` in the CR. + +**3. HPA lifecycle** — The operator creates, updates, and deletes the HPA as an owned resource tied to the FeatureStore CR. Removing the `autoscaling` configuration automatically cleans up the HPA. + +The scaling status is reported back on the FeatureStore status: + +```yaml +status: + scalingStatus: + currentReplicas: 3 + desiredReplicas: 3 +``` + +# What About TLS, CronJobs, and Services? + +Scaling is designed to work seamlessly with existing operator features: + +- **TLS** — Each pod mounts the same TLS secret. OpenShift service-serving certificates work automatically since they're bound to the Service, not individual pods. +- **Kubernetes Services** — The Service's label selector already matches all pods in the Deployment, so load balancing across replicas works out of the box. +- **CronJobs** — The `feast apply` and `feast materialize-incremental` CronJobs use `kubectl exec` into a single pod. Since DB-backed persistence is required for scaling, all pods share the same state — it doesn't matter which pod the CronJob runs against. + +# Getting Started + +**1. Ensure DB-backed persistence** for all enabled services (online store, offline store, registry). + +**2. Configure scaling** in your FeatureStore CR — use either static replicas or HPA (mutually exclusive): + +```yaml +spec: + replicas: 3 # static replicas (top-level) + # -- OR -- + # services: + # scaling: + # autoscaling: # HPA + # minReplicas: 2 + # maxReplicas: 10 +``` + +**3. Apply** the updated CR: + +```bash +kubectl apply -f my-featurestore.yaml +``` + +**4. Verify** the scaling: + +```bash +# Check pods +kubectl get pods -l app.kubernetes.io/managed-by=feast + +# Check HPA (if using autoscaling) +kubectl get hpa + +# Check FeatureStore status +kubectl get feast -o yaml +``` + +# Learn More + +- [Scaling Feast documentation](https://docs.feast.dev/how-to-guides/scaling-feast) +- [Feast on Kubernetes guide](https://docs.feast.dev/how-to-guides/feast-on-kubernetes) +- [FeatureStore CRD API reference](https://github.com/feast-dev/feast/blob/master/infra/feast-operator/docs/api/markdown/ref.md) +- [Sample CRs for static scaling and HPA](https://github.com/feast-dev/feast/tree/master/infra/feast-operator/config/samples) +- Join the [Feast Slack](https://slack.feast.dev) to share feedback and ask questions + +We're excited to see teams scale their feature serving infrastructure with confidence. Try it out and let us know how it works for your use case!