I think we dont need to provide the materialize_mode config. We should strictly feature server take care of materialization delegation to compute engine. Leaving no choice to do materialization on client side.

A cleaner approach might be: a RemoteComputeEngine (like RemoteOnlineStore/RemoteOfflineStore) that POSTs to the existing /materialize endpoint, returning a job handle with a dedicated /materialize-status/{job_id} endpoint - keeping the materialization delegation at the engine layer rather than the FeatureStore layer.

if you mean something like

batch_engine:
  type: remote.engine
  path: https://feast-online:443   # same host as online
  cert: /var/run/secrets/.../service-ca.crt

I thought about this before choosing current approach. I see two points which made me not take this decision.

1. This adds another config section which has to indicate only one thing, i.e. trigger materialization on feature server. For this, it add 1 config section & user has to configure 2 redundant fields.
2. Once client is initialized, user wont have option to try normal materialization flow if remote materialization fails for any reason.

My proposed approach is

store.materialize(remote=True, wait=False) --> pre-process check & return 202 Accepted response
store.poll_materialization() --> returns status

This approach has

1. Backward compatibility - User can fall back without re-initializing client.
2. Flexibility of Status polling - Full control to user to set interval of polling.
3. No need of Operator changes.
4. The same API can work for UI as well

@aniketpalu Ok, so, I think this PR is mixing three orthogonal concerns: async execution, remote delegation, and job tracking.

• Remote delegation already works today - POST /materialize runs materialization server-side. A client with online_store: remote can call this endpoint directly. As @jyejare mentioned, the feature server should strictly handle materialization - no per-call remote=True flag needed; the deployment topology already decides this.

• Async execution - The real gap is that long-running engines (SparkApplication) block the HTTP caller. The existing /materialize endpoint just needs an async mode (?async=true → 202 with job_id) rather than entirely new /materialize-async endpoints. This serves both UI on-demand requests and SDK wait=False use cases, while CronJob continues using the existing sync path unchanged.

• Job tracking - Polling FeatureViewState from the registry is fragile (race conditions, no job_id, ambiguous failure, breaks FeatureView.eq). If we need job tracking, it should be a dedicated MaterializationJob entity in the registry with proper RPCs - not piggybacking on FV metadata. For SparkApplication, the SparkApp CR status on K8s is already the source of truth.

May be we can scope this PR to just remote delegation and async execution ? ?async=true on the existing /materialize endpoint (fire-and-forget, returns 202. Completion is already observable via existing mechanisms like materialization_intervals on the FV, SparkApp CR status) + SDK support for async with remote delegation.

Job tracking with dedicated status endpoints / registry entities can come as a separate effort.

Remote delegation already works today - POST /materialize runs materialization server-side. A client with online_store: remote can call this endpoint directly. As @jyejare mentioned, the feature server should strictly handle materialization - no per-call remote=True flag needed; the deployment topology already decides this.
Async execution - The real gap is that long-running engines (SparkApplication) block the HTTP caller. The existing /materialize endpoint just needs an async mode (?async=true → 202 with job_id) rather than entirely new /materialize-async endpoints. This serves both UI on-demand requests and SDK wait=False use cases, while CronJob continues using the existing sync path unchanged.

agree on both point

[Critical] Function defined inside endpoint handler scope

The function _authorize_materialize_views is defined inside the endpoint handler scope but references store from the outer scope. This creates a closure that captures the store variable, which could lead to unexpected behavior if the store changes or if this function is called in different contexts. This pattern violates the principle of explicit dependencies and makes testing difficult.

Suggested:

[Critical] Function defined inside endpoint handler with closure dependencies

Similar to the previous issue, _check_already_materializing and other helper functions are defined inside the endpoint handler and capture variables from outer scope. This makes the code harder to test and reason about. These functions should be extracted as proper module-level functions with explicit parameters.

Suggested:

[Warning] Materialize async endpoint lacks proper error handling

The /materialize-async endpoint catches all exceptions in the background thread but only logs them. If the materialization fails during the initial setup (before the background thread), the state won't be properly reset. The endpoint should have a try-catch around the initial setup and reset states appropriately.

Suggested:

[Warning] Empty return in state transition function breaks existing logic

Adding an early return when _force_local=True means the previous_states[feature_view.name] = previous_state line is never reached, which could break rollback logic elsewhere. The function should still record the previous state even when skipping the transition.

Suggested:

[Warning] State mapping logic could be more explicit

The _fv_state_to_materialization_status function uses a mutable seen_materializing set that's passed by reference. This side-effect pattern makes the function harder to reason about and test. Consider returning a tuple or using a more functional approach.

Suggested:

[Warning] HTTP session not properly closed in all paths

The HTTP session is closed in a finally block, but if an exception occurs during session creation or configuration, the session might never be assigned, leading to a NameError. The session creation and closure should be more robust.

Suggested: