Commit 194555c

(or equivalent):

{% page-ref page="spark_application.md" %}

- Batch materialization via Kubeflow Spark Operator `SparkApplication` CRs

- One SparkApplication per materialize call (multi–feature-view batching)

- Requires network-accessible online/offline/registry stores (no file-based backends)

The **SparkApplication** compute engine runs Feast **batch materialization** on Kubernetes by creating a [Kubeflow Spark Operator](https://github.com/kubeflow/spark-operator) `SparkApplication` custom resource for each materialization job.

Unlike the in-process [`spark.engine`](spark.md) compute engine (which uses a Spark session inside the Feast process), `spark_application` submits work to the Spark Operator. The operator starts a driver pod and executors from your configured image; Feast polls the SparkApplication until it completes.

| Capability | Supported |

|------------|-----------|

| `materialize` / `materialize-incremental` | Yes |

| Multiple feature views in one job | Yes — one SparkApplication per materialize call |

| `get_historical_features` | Not yet |

| SparkConnect | Separate approach — not this engine |

1. Feast creates a ConfigMap with job tasks and a driver copy of `feature_store.yaml`.

2. Feast creates a `SparkApplication` CR pointing at the driver entrypoint (`main.py` in the image).

3. Inside the pod, the batch engine type is rewritten to `spark.engine` so materialization uses the Spark session created by `spark-submit` (avoids recursive SparkApplication creation).

4. The driver writes features to your configured **online store** and updates the **registry** (same network backends as the server).

- Kubeflow Spark Operator installed and watching the target namespace.

- A container **image** that includes the Feast SDK, PySpark, and clients for your stores. See the reference [Dockerfile](https://github.com/feast-dev/feast/blob/master/sdk/python/feast/infra/compute_engines/spark_application/Dockerfile).

- **Network-accessible** online store, offline store, and registry. File-based backends are rejected because Spark pods have an ephemeral filesystem:

| Rejected | Examples | Use instead |

|----------|----------|-------------|

| File online | `sqlite`, `faiss` | Redis, remote online, etc. |

| File offline | `dask`, `file`, `duckdb` | `spark`, Postgres, Snowflake, BigQuery, etc. |

| File registry | `file` | SQL registry, Snowflake |

For distributed reads, configure `offline_store.type: spark` (or another store Spark can read efficiently).

When using the Feast Operator:

- Point `spec.batchEngine.configMapRef` at a ConfigMap whose `type` is `spark_application` (see [Guide 6 — Batch Engine & Scheduled Jobs](../../how-to-guides/feast-operator/06-batch-and-jobs.md)).

- The operator auto-creates RBAC for the `spark_application` batch engine (server and driver service accounts).

- Set `spec.services.initImage` if init / `feast-apply` containers need the Spark-capable image.

{% code title="feature_store.yaml" %}

project: my_project

registry:

registry_type: sql

path: postgresql+psycopg://feast:****@postgres:5432/feast

online_store:

type: redis

connection_string: redis:6379

offline_store:

type: spark

spark_conf:

spark.master: local[*]

batch_engine:

type: spark_application

image: my-registry.example.com/feast-spark-driver:latest

namespace: feast

spark_version: "4.0.1"

driver_cores: 1

driver_memory: "2g"

executor_instances: 2

executor_cores: 1

executor_memory: "2g"

spark_conf:

spark.sql.shuffle.partitions: "100"

apiVersion: v1

kind: ConfigMap

metadata:

name: feast-spark-batch-engine

namespace: feast

data:

config: |

---

apiVersion: feast.dev/v1

kind: FeatureStore

metadata:

name: feast

namespace: feast

spec:

feastProject: my_project

batchEngine:

configMapRef:

name: feast-spark-batch-engine

configMapKey: config

- Default (`run_async=False`): block until the server finishes sync materialization.

- `run_async=True`: accept asynchronously (`?async=true`); poll feature-view state in the registry for completion.

- `force=True` (with `run_async=True`): override stuck `MATERIALIZING` state on the server.

---

---

---

- [Spark compute engine (in-process)](spark.md)

- [Feast Operator — batch engine ConfigMap](../../how-to-guides/feast-operator/06-batch-and-jobs.md)

- [Creating a custom compute engine](../../how-to-guides/customizing-feast/creating-a-custom-compute-engine.md)

from typing import TYPE_CHECKING, Any, Dict, List, Optional, Union

from torch import Tensor as TorchTensor

TorchTensor = Any

from feast.errors import FeastObjectNotFoundException, FeastPermissionError

get_security_manager,

is_auth_necessary,

# Per-object mutations are authorized in Apply*/Delete* RPCs.

# Requiring UPDATE on every project breaks shared-registry multi-tenant

# commits (remote feastRef with different feastProjects). Require CREATE

# or UPDATE on at least one project when auth is enabled instead.

projects = cast(

list[FeastObject],

list(self.proxied_registry.list_projects(allow_cache=True)),

)

if projects and is_auth_necessary(get_security_manager()):

can_update = permitted_resources(

resources=projects, actions=AuthzedAction.UPDATE

)

can_create = permitted_resources(

resources=projects, actions=AuthzedAction.CREATE

)

if not can_update and not can_create:

raise FeastPermissionError(

"Not authorized to commit registry changes: "

"CREATE or UPDATE permission required on at least one project"

)

# Use create-or-update authorization so first apply of a new project over

# a remote registry can refresh before the project exists yet.

assert_permissions_to_update(

resource=Project(name=request.project),

getter=self.proxied_registry.get_project,

project=request.project,