feat(cassandra): Add multi-DC support via per-datacenter execution profiles by singhhimanshu0811 · Pull Request #6434 · feast-dev/feast

singhhimanshu0811 · 2026-05-24T18:20:02Z

##Which issue(s) this PR fixes
Fixes #6433

Summary

Adds a datacenters config list to CassandraOnlineStoreConfig as an alternative to the flat hosts field, enabling proper multi-datacenter support.

Each datacenter entry gets a named Cassandra execution profile (keyed by local_dc), so pipeline code can route reads/writes to a specific DC by passing execution_profile="<dc-name>" to the driver. The default profile is pinned to load_balancing.local_dc, or the first DC when load_balancing is absent.

Changes

New CassandraDatacenterConfig inner model with local_dc, hosts, replication_factor (informational), replication_strategy (informational)
New datacenters: Optional[List[CassandraDatacenterConfig]] field on CassandraOnlineStoreConfig
New Branch B in _get_session() that builds per-DC execution profiles and connects with all DC hosts merged; original hosts/secure_bundle_path path (Branch A) is completely untouched
_dc_execution_profiles: List[str] attribute exposes registered profile names
Updated docs/reference/online-stores/cassandra.md with a multi-DC feature_store.yaml example

Motivation

The flat hosts + single local_dc model forces all hosts to belong to the same datacenter. Users with multi-DC clusters had to create separate feature store projects per DC. With this change:

A single feature store config can span multiple DCs
The Cassandra driver routes to the correct DC via execution profiles
Per-DC DCAwareRoundRobinPolicy prevents reads from spilling into unintended regions
Advanced pipelines can explicitly target a DC (e.g. read from the region where data has already been replicated)

Backward compatibility

Fully backward compatible — datacenters is optional and the original code path is untouched.

ntkathole

Review of PR #6434 — Multi-DC Cassandra Support

Thanks for the contribution, @singhhimanshu0811! This directly addresses the gap we discussed around single-DC limitations in the Cassandra online store config. Overall the approach is solid — using per-DC execution profiles is the right driver-level mechanism. I have several items ranging from bugs to design suggestions:

1. Bug: Mutable class-level default for `_dc_execution_profiles`

_dc_execution_profiles: List[str] = []

This is a mutable class attribute shared across all instances. If multiple CassandraOnlineStore instances are created (e.g., in tests or multi-project setups), they'll all append to the same list. This same pattern issue exists for _prepared_statements in the original code, but introducing a new one should be avoided.

Fix: Initialize in _get_session or use a property pattern:

_dc_execution_profiles: List[str]

def _get_session(self, config: RepoConfig):
    ...
    if online_store_config.datacenters:
        self._dc_execution_profiles = []  # reset per connection
        ...

2. Bug: Missing mutual exclusivity validation

The docstring says:

Mutually exclusive with hosts and secure_bundle_path.

But there's no actual validation enforcing this. If a user sets both datacenters and hosts, Branch B runs (because it checks datacenters first) and silently ignores hosts. This will confuse users.

Fix: Add a Pydantic model_validator or an explicit check at the top of _get_session:

if online_store_config.datacenters and (
    online_store_config.hosts or online_store_config.secure_bundle_path
):
    raise CassandraInvalidConfig(
        "Cassandra config: 'datacenters' is mutually exclusive with "
        "'hosts' and 'secure_bundle_path'"
    )

3. Design: Profiles are created but never used by Feast operations

The PR creates named execution profiles and exposes them via _dc_execution_profiles, but none of the existing Feast operations (online_read, online_write_batch, update) pass execution_profile= to the driver. All queries will use EXEC_PROFILE_DEFAULT — effectively still single-DC behavior.

This means:

The profiles are registered but dead code from Feast's perspective
Users can't actually route reads/writes to a specific DC without modifying Feast internals

I understand this may be intended as a foundational PR with follow-up work to add target_dc parameters to the Feast API. If so, please mention this explicitly in the PR description and/or open a tracking issue for the follow-up. Otherwise, consider adding at minimum an optional execution_profile pass-through in _read_rows_by_entity_keys and _write_rows_concurrently.

4. Design: `replication_factor` and `replication_strategy` are purely informational

These fields are documented as "informational" and Feast doesn't use them. This raises questions:

If they're never consumed by code, why are they in the config? They'll confuse users who expect Feast to act on them.
If the intent is future keyspace creation support, consider deferring these fields to that PR to keep this one focused.

At minimum, add a log warning if these are set:

if dc.replication_factor or dc.replication_strategy:
    logger.info(
        "Cassandra multi-DC: replication_factor/replication_strategy are "
        "informational only; keyspace must be pre-created."
    )

5. Suggestion: Duplicate `local_dc` naming is confusing

In the config structure, local_dc appears in two places with different semantics:

datacenters[*].local_dc — the datacenter name (identifier)
load_balancing.local_dc — which DC is the default

Consider renaming the per-DC field to just name or dc_name for clarity:

datacenters:
  - name: dc1        # clearer than local_dc
    hosts: [...]
  - name: dc2
    hosts: [...]
load_balancing:
    local_dc: dc1    # this selects the default

The field is called local_dc because that's what the driver's DCAwareRoundRobinPolicy parameter is named, but at the config level it reads as "this datacenter's local_dc" which is semantically odd.

6. Suggestion: Empty `datacenters` list edge case

What happens if datacenters: [] (empty list)? The truthiness check if online_store_config.datacenters: will be False, so it falls through to Branch A. But Branch A will then fail with E_CASSANDRA_NOT_CONFIGURED because hosts is also None. The error message will be misleading.

Fix: Validate that datacenters contains at least one entry if set:

if online_store_config.datacenters is not None and len(online_store_config.datacenters) == 0:
    raise CassandraInvalidConfig("Cassandra 'datacenters' list must not be empty")

7. Tests needed

There are no unit tests for the new code path. Please add tests covering:

Config parsing with datacenters field
Mutual exclusivity validation (once added)
Default DC selection (from load_balancing.local_dc vs. first-in-list fallback)
Error case: load_balancing.local_dc doesn't match any DC entry
Integration with the existing CassandraOnlineStoreCreator test fixture (even if it's a single-DC test, it validates backward compat)

8. Minor: Redundant `if not self._session:` in Branch A

After Branch B was inserted, the existing code still has:

if not self._session:
    # configuration consistency checks
    ...

This if not self._session: is now always True when reached (because we already returned early at line 294 if self._session: return). It's harmless but misleading — the original code had it as the single branch, now it's dead logic. Consider removing the if wrapper for clarity (keeping the body).

Summary

Item	Severity	Action needed
Mutable class-level list	Bug	Must fix
Missing mutual exclusivity check	Bug	Must fix
Profiles unused by Feast ops	Design gap	Document plan or implement
Informational-only fields	Design	Consider removing or add warning
`local_dc` naming confusion	Suggestion	Consider rename
Empty list edge case	Suggestion	Add validation
No tests	Required	Add unit tests
Redundant `if not self._session`	Minor	Cleanup

The core idea is good and aligns with a real production need. Looking forward to the next iteration!

singhhimanshu0811 · 2026-05-25T11:15:15Z

Hey @ntkathole thanks for replying back. I'll revert back with updated pr in 1-2 days, meanwhile could you review these once
for point 1 , I was wondering if we could solve the prepared statement with this pr only ,
for 3, i was planning to do it later, but now that I have some time, I can resolve in this pr only. will do this.
for 4, i ve had few questions, why is there no provision to create keyspace in the code? perhaps we can have a _create_keyspace function?
The rest, are great points, will resolve them in new push.

singhhimanshu0811 · 2026-05-25T13:03:06Z

@ntkathole for solving issue 3, basically incorporating profiles in read and write, i am thinking something like this

online_store:
type: cassandra
keyspace: feast_keyspace
datacenters:
- name: dc1
hosts: [192.168.1.1, 192.168.1.2]
- name: dc2
hosts: [10.0.0.1]
routing:
read_dc: dc2 # reads go to dc2
write_dc: dc1 # writes go to dc1

basically along with datacenters, adding a routing header, so user can decide at start time, that in this feature store session, which dc they would like to read from or which dc they would like to write from. does this work?

and of course there would be validation that routing header params are legal, i.e in datacenter header params

ntkathole · 2026-05-26T14:56:41Z

@singhhimanshu0811

I would recommend making _create_keyspace a separate PR.
For routing config - yes, I think that covers common use cases, - Make both read_dc and write_dc fields optional with a default value.

ntkathole · 2026-06-09T03:49:18Z

@singhhimanshu0811 Please fix linting here as well

franciscojavierarceo · 2026-06-17T03:23:14Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4d21842068

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

singhhimanshu0811 · 2026-06-28T07:08:54Z

hi @ntkathole @franciscojavierarceo could you kindly review this pr once , could this be merged if there are no pending issues? Thanks!!Please let me know in case of some issue!

ntkathole · 2026-06-28T07:36:21Z

@singhhimanshu0811 Need to regenerate pixi.lock file since pyproject.toml is changed, otherwise looks good to me

…ofiles Introduce a `datacenters` config list as an alternative to the flat `hosts` field. Each entry specifies the contact points, local_dc, and optional replication metadata for one datacenter. On connect, a named Cassandra execution profile is registered per DC (profile key = local_dc), enabling callers to route driver-level queries to a specific datacenter. The default profile is pinned to `load_balancing.local_dc`, or the first DC when load_balancing is absent. All existing `hosts` / `secure_bundle_path` configs are fully backward compatible — the original code path is untouched. Also updates docs/reference/online-stores/cassandra.md with a multi-DC feature_store.yaml example. Signed-off-by: singhhimanshu0811 email : itssinghhimanshu@gmail.com Signed-off-by: Himanshu Singh <himanshu.singh@walmart.com>

Signed-off-by: Himanshu Singh <himanshu.singh@walmart.com>

Signed-off-by: Himanshu Singh <itssinghhimanshu@gmail.com> Signed-off-by: Himanshu Singh <himanshu.singh@walmart.com>

Signed-off-by: Himanshu Singh <himanshu.singh@walmart.com>

…pport in execute_concurrent_with_args Signed-off-by: Himanshu Singh <himanshu.singh@walmart.com>

…kwargs Signed-off-by: Himanshu Singh <himanshu.singh@walmart.com>

…ef error Signed-off-by: Himanshu Singh <himanshu.singh@walmart.com>

Signed-off-by: Himanshu Singh <himanshu.singh@walmart.com>

singhhimanshu0811 · 2026-06-28T08:00:55Z

Hey @ntkathole i regenarted pixi, rebased and pushed again. can you review

# [0.65.0](v0.64.0...v0.65.0) (2026-07-20) ### Bug Fixes * add debug logging for FIPS mode detection fallback ([6c1b24e](6c1b24e)) * Build embedded UI from local source ([#6525](#6525)) ([3500349](3500349)) * Bump decommissioned Snowflake Python UDF runtime from 3.9 to 3.10 ([#6606](#6606)) ([#6608](#6608)) ([10341e4](10341e4)) * configure FIPS-compliant gRPC cipher suites for offline server ([6bc80a2](6bc80a2)) * Correct Flink PyArrow dependency constraints ([#6604](#6604)) ([70a9751](70a9751)) * Fix ValueError in signal handling for Trino worker threads ([#6428](#6428)) ([506d919](506d919)) * Fixed monitoring page issues ([7946018](7946018)) * Make pytest config compatible with newer pytest ([#5779](#5779)) ([a57ea33](a57ea33)) * Replace comma with space in DynamoDB-incompatible label tag value ([51e3a16](51e3a16)) * Resolve UI build warnings ([#6529](#6529)) ([abe92af](abe92af)) * Unblock nightly UI build ([#6570](#6570)) ([f296d4b](f296d4b)) * Use LONGBLOB for SQL registry proto columns on MySQL ([#6566](#6566)) ([7e4beb2](7e4beb2)) ### Features * Add click-to-zoom lightbox for blog post images ([#6575](#6575)) ([1cb23fd](1cb23fd)) * Add dark mode support to website and blog ([#6589](#6589)) ([7358fb8](7358fb8)) * Add OnlineStore for Aerospike ([#6532](#6532)) ([9cd35e1](9cd35e1)) * Add OpenLineage Consumer to Feast - receive, store, and visualize cross-producer lineage ([#6549](#6549)) ([a834126](a834126)) * Add registry list feature views by updated since ([#6092](#6092)) ([#6093](#6093)) ([006c606](006c606)) * Add ScyllaDB online store with vector search ([#6508](#6508)) ([1669661](1669661)) * Added compute and jobs UI ([ba2c05c](ba2c05c)) * Added Iceberg REST Catalog data source support ([e0a8573](e0a8573)) * Bring Your Own Spark - SparkApplication ([#6550](#6550)) ([dcd496f](dcd496f)) * **cassandra:** Add multi-DC support via per-datacenter execution profiles ([#6434](#6434)) ([0de9196](0de9196)) * Enhanced data source creation as a visual catalog with type-specific forms ([#6557](#6557)) ([d6acbba](d6acbba)) * Enhanced datasets UI functionality ([de11152](de11152)) * Implement RegistryServer.Proto RPC with RBAC-filtered response ([#6558](#6558)) ([#6552](#6552)) ([0d02614](0d02614)) * New zoned timestamp feature type ([#6536](#6536)) ([#6537](#6537)) ([eb042f0](eb042f0)) * **operator:** Auto-create RBAC for spark_application batch engine ([#6597](#6597)) ([f487b37](f487b37)) * **operator:** integrate cluster TLS profile for OCP 5.0 compliance ([43263a6](43263a6)) * Permissions CRUD UI and OIDC auth integration in UI ([6511da1](6511da1)) * Retrieve historical features from BigQuery without entity_df ([#6569](#6569)) ([cd5f6bb](cd5f6bb)), closes [#6558](#6558) [#6552](#6552) * **spark:** SparkSource query+path and pre-computed offline read for BatchFeatureView ([#6440](#6440)) ([4dc8757](4dc8757)) ### BREAKING CHANGES * total_timeout_ms is renamed to batch_total_timeout_ms. Config files using the old name must be updated. No default value change. Docs updated (reference + perf-tuning guide) with a short explainer on the per-attempt vs total deadline distinction. Two new unit tests pin the policy wiring: socket_timeout_ms propagates to all three scopes, and is omitted (not injected as None) when unset. Signed-off-by: Valentyn Kahamlyk <valentin.kagamlyk@gmail.com> * refactor(aerospike): use MAP_KEY_ORDERED, KEY_DIGEST, and instance-scoped client Cheap-win cleanups flagged in review, all touching the same small patch of write-path and lifecycle code. * Map CDTs are now created with MAP_KEY_ORDERED. map_get_by_key / map_remove_by_key on an ordered map are O(log N) in the map size instead of O(N); matters on reads of wide feature views and on the update() background scan (which walks every record in the project's set). * Writes drop POLICY_KEY_SEND and rely on the client default (POLICY_KEY_DIGEST). The serialized entity key is no longer stored alongside each record, saving per-record storage the read path never consumes (batch_operate preserves request order; results are paired back by zip in online_read). * _client moves from a class attribute to an instance attribute (set in __init__). Previously two AerospikeOnlineStore instances could share the cached client through class state until one wrote self._client. With the instance attribute the state is always per-instance from construction. * Drop MongoDB references from class docstrings and comments (they referred to how the storage layout was derived rather than documenting current behavior). Also rewrite the _build_batch_writes docstring to describe the policies applied on the write path. Unit test assertions for the write-path record are updated: bw.policy is now None (client default applies) and map ops carry map_policy={'map_order': MAP_KEY_ORDERED}. All three docker-backed integration tests still pass end-to-end (cross-FV upsert, update() background scan, full feature-store round-trip), so the read/write shape survives the ordering and policy changes against a real server. Signed-off-by: Valentyn Kahamlyk <valentin.kagamlyk@gmail.com> * feat(aerospike): add per-FV namespace/set overrides and prewriting hook Adds three configuration knobs to AerospikeOnlineStoreConfig: - namespace_overrides: pin individual feature views to a different Aerospike namespace (e.g. RAM-only vs. SSD-backed) without splitting the project across stores. - set_overrides: place a feature view in its own set so admin ops on it (truncate, scan-based deletes during `feast apply`) do not touch records of other views. - prewriting_hook: import-string-resolved callable invoked once per online_write_batch with the rows about to be written, returning the rows that actually go on the wire. Resolved and cached on first use; returning [] short-circuits the wire call. Read, write, update and teardown paths all honour the per-FV ns/set resolution. update() groups dropped feature views by their resolved (ns, set) pair and issues one background scan per group. teardown() truncates every unique (ns, set) pair the project may have written to, including the store-level default. Adds 22 unit tests for the new behaviour and updates 3 existing call sites of _build_batch_writes for the new namespace= parameter. Adds a sample hook module under examples/online_store/aerospike_overrides_and_hooks/ and corresponding sections in docs/reference/online-stores/aerospike.md. Signed-off-by: Valentyn Kahamlyk <valentin.kagamlyk@gmail.com> * test: update aerospike image tag Signed-off-by: Valentyn Kahamlyk <valentin.kagamlyk@gmail.com> * chore: sync README template and secrets baseline after master merge Signed-off-by: Valentyn Kahamlyk <valentin.kagamlyk@gmail.com> * chore: fix secrets baseline line number for v1 operator types Adding aerospike to the feast-operator enum shifted the allowlisted SecretRef entry in api/v1/featurestore_types.go by one line. Signed-off-by: Valentyn Kahamlyk <valentin.kagamlyk@gmail.com> * docs: update aerospike docs Signed-off-by: Valentyn Kahamlyk <valentin.kagamlyk@gmail.com> * fix(aerospike): wire batch max_retries and fix empty projection handling Copilot review feedback on PR #6532: - Add max_retries to the batch client policy (batch_operate/batch_write path) - Treat empty projected feature maps as present FV slots (is not None) - Return {} from _normalize_projected_features([]) instead of None - Fix projection unit test mock/assertions - Correct prewriting_hook config docstring Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Valentyn Kahamlyk <valentin.kagamlyk@gmail.com> * style(aerospike): format online_read docs assignment for ruff Signed-off-by: Valentyn Kahamlyk <valentin.kagamlyk@gmail.com> * chore: update pixi.lock for aerospike optional extra Regenerate the v6 lockfile with Pixi v0.63.1 after adding the aerospike extra to pyproject.toml. Signed-off-by: Valentyn Kahamlyk <valentin.kagamlyk@gmail.com> * fix(aerospike): add client init lock and batch chunking Guard lazy client creation with a lock to avoid connection leaks under concurrent first use, and chunk batch reads/writes by batch_max_records so large materializations stay under Aerospike server batch limits. Signed-off-by: Valentyn Kahamlyk <valentin.kagamlyk@gmail.com>

# [0.65.0](feast-dev/feast@v0.64.0...v0.65.0) (2026-07-20) ### Bug Fixes * add debug logging for FIPS mode detection fallback ([6c1b24e](feast-dev@6c1b24e)) * Build embedded UI from local source ([feast-dev#6525](feast-dev#6525)) ([3500349](feast-dev@3500349)) * Bump decommissioned Snowflake Python UDF runtime from 3.9 to 3.10 ([feast-dev#6606](feast-dev#6606)) ([feast-dev#6608](feast-dev#6608)) ([10341e4](feast-dev@10341e4)) * configure FIPS-compliant gRPC cipher suites for offline server ([6bc80a2](feast-dev@6bc80a2)) * Correct Flink PyArrow dependency constraints ([feast-dev#6604](feast-dev#6604)) ([70a9751](feast-dev@70a9751)) * Fix ValueError in signal handling for Trino worker threads ([feast-dev#6428](feast-dev#6428)) ([506d919](feast-dev@506d919)) * Fixed monitoring page issues ([7946018](feast-dev@7946018)) * Make pytest config compatible with newer pytest ([feast-dev#5779](feast-dev#5779)) ([a57ea33](feast-dev@a57ea33)) * Replace comma with space in DynamoDB-incompatible label tag value ([51e3a16](feast-dev@51e3a16)) * Resolve UI build warnings ([feast-dev#6529](feast-dev#6529)) ([abe92af](feast-dev@abe92af)) * Unblock nightly UI build ([feast-dev#6570](feast-dev#6570)) ([f296d4b](feast-dev@f296d4b)) * Use LONGBLOB for SQL registry proto columns on MySQL ([feast-dev#6566](feast-dev#6566)) ([7e4beb2](feast-dev@7e4beb2)) ### Features * Add click-to-zoom lightbox for blog post images ([feast-dev#6575](feast-dev#6575)) ([1cb23fd](feast-dev@1cb23fd)) * Add dark mode support to website and blog ([feast-dev#6589](feast-dev#6589)) ([7358fb8](feast-dev@7358fb8)) * Add OnlineStore for Aerospike ([feast-dev#6532](feast-dev#6532)) ([9cd35e1](feast-dev@9cd35e1)) * Add OpenLineage Consumer to Feast - receive, store, and visualize cross-producer lineage ([feast-dev#6549](feast-dev#6549)) ([a834126](feast-dev@a834126)) * Add registry list feature views by updated since ([feast-dev#6092](feast-dev#6092)) ([feast-dev#6093](feast-dev#6093)) ([006c606](feast-dev@006c606)) * Add ScyllaDB online store with vector search ([feast-dev#6508](feast-dev#6508)) ([1669661](feast-dev@1669661)) * Added compute and jobs UI ([ba2c05c](feast-dev@ba2c05c)) * Added Iceberg REST Catalog data source support ([e0a8573](feast-dev@e0a8573)) * Bring Your Own Spark - SparkApplication ([feast-dev#6550](feast-dev#6550)) ([dcd496f](feast-dev@dcd496f)) * **cassandra:** Add multi-DC support via per-datacenter execution profiles ([feast-dev#6434](feast-dev#6434)) ([0de9196](feast-dev@0de9196)) * Enhanced data source creation as a visual catalog with type-specific forms ([feast-dev#6557](feast-dev#6557)) ([d6acbba](feast-dev@d6acbba)) * Enhanced datasets UI functionality ([de11152](feast-dev@de11152)) * Implement RegistryServer.Proto RPC with RBAC-filtered response ([feast-dev#6558](feast-dev#6558)) ([feast-dev#6552](feast-dev#6552)) ([0d02614](feast-dev@0d02614)) * New zoned timestamp feature type ([feast-dev#6536](feast-dev#6536)) ([feast-dev#6537](feast-dev#6537)) ([eb042f0](feast-dev@eb042f0)) * **operator:** Auto-create RBAC for spark_application batch engine ([feast-dev#6597](feast-dev#6597)) ([f487b37](feast-dev@f487b37)) * **operator:** integrate cluster TLS profile for OCP 5.0 compliance ([43263a6](feast-dev@43263a6)) * Permissions CRUD UI and OIDC auth integration in UI ([6511da1](feast-dev@6511da1)) * Retrieve historical features from BigQuery without entity_df ([feast-dev#6569](feast-dev#6569)) ([cd5f6bb](feast-dev@cd5f6bb)), closes [feast-dev#6558](feast-dev#6558) [feast-dev#6552](feast-dev#6552) * **spark:** SparkSource query+path and pre-computed offline read for BatchFeatureView ([feast-dev#6440](feast-dev#6440)) ([4dc8757](feast-dev@4dc8757)) ### BREAKING CHANGES * total_timeout_ms is renamed to batch_total_timeout_ms. Config files using the old name must be updated. No default value change. Docs updated (reference + perf-tuning guide) with a short explainer on the per-attempt vs total deadline distinction. Two new unit tests pin the policy wiring: socket_timeout_ms propagates to all three scopes, and is omitted (not injected as None) when unset. Signed-off-by: Valentyn Kahamlyk <valentin.kagamlyk@gmail.com> * refactor(aerospike): use MAP_KEY_ORDERED, KEY_DIGEST, and instance-scoped client Cheap-win cleanups flagged in review, all touching the same small patch of write-path and lifecycle code. * Map CDTs are now created with MAP_KEY_ORDERED. map_get_by_key / map_remove_by_key on an ordered map are O(log N) in the map size instead of O(N); matters on reads of wide feature views and on the update() background scan (which walks every record in the project's set). * Writes drop POLICY_KEY_SEND and rely on the client default (POLICY_KEY_DIGEST). The serialized entity key is no longer stored alongside each record, saving per-record storage the read path never consumes (batch_operate preserves request order; results are paired back by zip in online_read). * _client moves from a class attribute to an instance attribute (set in __init__). Previously two AerospikeOnlineStore instances could share the cached client through class state until one wrote self._client. With the instance attribute the state is always per-instance from construction. * Drop MongoDB references from class docstrings and comments (they referred to how the storage layout was derived rather than documenting current behavior). Also rewrite the _build_batch_writes docstring to describe the policies applied on the write path. Unit test assertions for the write-path record are updated: bw.policy is now None (client default applies) and map ops carry map_policy={'map_order': MAP_KEY_ORDERED}. All three docker-backed integration tests still pass end-to-end (cross-FV upsert, update() background scan, full feature-store round-trip), so the read/write shape survives the ordering and policy changes against a real server. Signed-off-by: Valentyn Kahamlyk <valentin.kagamlyk@gmail.com> * feat(aerospike): add per-FV namespace/set overrides and prewriting hook Adds three configuration knobs to AerospikeOnlineStoreConfig: - namespace_overrides: pin individual feature views to a different Aerospike namespace (e.g. RAM-only vs. SSD-backed) without splitting the project across stores. - set_overrides: place a feature view in its own set so admin ops on it (truncate, scan-based deletes during `feast apply`) do not touch records of other views. - prewriting_hook: import-string-resolved callable invoked once per online_write_batch with the rows about to be written, returning the rows that actually go on the wire. Resolved and cached on first use; returning [] short-circuits the wire call. Read, write, update and teardown paths all honour the per-FV ns/set resolution. update() groups dropped feature views by their resolved (ns, set) pair and issues one background scan per group. teardown() truncates every unique (ns, set) pair the project may have written to, including the store-level default. Adds 22 unit tests for the new behaviour and updates 3 existing call sites of _build_batch_writes for the new namespace= parameter. Adds a sample hook module under examples/online_store/aerospike_overrides_and_hooks/ and corresponding sections in docs/reference/online-stores/aerospike.md. Signed-off-by: Valentyn Kahamlyk <valentin.kagamlyk@gmail.com> * test: update aerospike image tag Signed-off-by: Valentyn Kahamlyk <valentin.kagamlyk@gmail.com> * chore: sync README template and secrets baseline after master merge Signed-off-by: Valentyn Kahamlyk <valentin.kagamlyk@gmail.com> * chore: fix secrets baseline line number for v1 operator types Adding aerospike to the feast-operator enum shifted the allowlisted SecretRef entry in api/v1/featurestore_types.go by one line. Signed-off-by: Valentyn Kahamlyk <valentin.kagamlyk@gmail.com> * docs: update aerospike docs Signed-off-by: Valentyn Kahamlyk <valentin.kagamlyk@gmail.com> * fix(aerospike): wire batch max_retries and fix empty projection handling Copilot review feedback on PR feast-dev#6532: - Add max_retries to the batch client policy (batch_operate/batch_write path) - Treat empty projected feature maps as present FV slots (is not None) - Return {} from _normalize_projected_features([]) instead of None - Fix projection unit test mock/assertions - Correct prewriting_hook config docstring Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Valentyn Kahamlyk <valentin.kagamlyk@gmail.com> * style(aerospike): format online_read docs assignment for ruff Signed-off-by: Valentyn Kahamlyk <valentin.kagamlyk@gmail.com> * chore: update pixi.lock for aerospike optional extra Regenerate the v6 lockfile with Pixi v0.63.1 after adding the aerospike extra to pyproject.toml. Signed-off-by: Valentyn Kahamlyk <valentin.kagamlyk@gmail.com> * fix(aerospike): add client init lock and batch chunking Guard lazy client creation with a lock to avoid connection leaks under concurrent first use, and chunk batch reads/writes by batch_max_records so large materializations stay under Aerospike server batch limits. Signed-off-by: Valentyn Kahamlyk <valentin.kagamlyk@gmail.com>

singhhimanshu0811 requested a review from a team as a code owner May 24, 2026 18:20

singhhimanshu0811 force-pushed the feat/cassandra-multi-dc-execution-profiles branch from 1ad4040 to 20539cf Compare May 24, 2026 18:24

ntkathole changed the title ~~feat(cassandra): add multi-DC support via per-datacenter execution profiles~~ feat(cassandra): Add multi-DC support via per-datacenter execution profiles May 25, 2026

ntkathole reviewed May 25, 2026

View reviewed changes

singhhimanshu0811 requested a review from ntkathole May 25, 2026 13:04

singhhimanshu0811 force-pushed the feat/cassandra-multi-dc-execution-profiles branch 2 times, most recently from 0511732 to d5947e9 Compare June 4, 2026 08:09

chatgpt-codex-connector Bot reviewed Jun 17, 2026

View reviewed changes

Comment thread sdk/python/feast/infra/online_stores/cassandra_online_store/cassandra_online_store.py

singhhimanshu0811 force-pushed the feat/cassandra-multi-dc-execution-profiles branch from 2f06c08 to 3a42466 Compare June 26, 2026 08:27

singhhimanshu0811 requested a review from franciscojavierarceo June 26, 2026 08:34

singhhimanshu0811 force-pushed the feat/cassandra-multi-dc-execution-profiles branch 2 times, most recently from 3353a99 to c27f407 Compare June 26, 2026 12:26

ntkathole added the ok-to-test label Jun 28, 2026

singhhimanshu0811 force-pushed the feat/cassandra-multi-dc-execution-profiles branch from 7405118 to a523814 Compare June 28, 2026 07:46

Himanshu Singh added 9 commits June 28, 2026 13:28

implemented suggestions and tests

180cede

Signed-off-by: Himanshu Singh <himanshu.singh@walmart.com>

doc -s

20d4328

Signed-off-by: Himanshu Singh <himanshu.singh@walmart.com>

doc -s

50bab2f

Signed-off-by: Himanshu Singh <himanshu.singh@walmart.com>

empty commit

fc3a315

Signed-off-by: Himanshu Singh <himanshu.singh@walmart.com>

trigger build

a1888b1

Signed-off-by: Himanshu Singh <itssinghhimanshu@gmail.com> Signed-off-by: Himanshu Singh <himanshu.singh@walmart.com>

secrets whitelisting

0b1238e

Signed-off-by: Himanshu Singh <himanshu.singh@walmart.com>

fix: apply ruff format and fix pre-commit secret/linting issues

9083426

Signed-off-by: Himanshu Singh <himanshu.singh@walmart.com>

fix: bump cassandra-driver minimum to 3.26.0 for execution_profile su…

8ddcc84

…pport in execute_concurrent_with_args Signed-off-by: Himanshu Singh <himanshu.singh@walmart.com>

Himanshu Singh added 6 commits June 28, 2026 13:28

fix: add type annotations to resolve mypy errors in Branch A cluster_…

3f12e15

…kwargs Signed-off-by: Himanshu Singh <himanshu.singh@walmart.com>

fix: rename Branch A execution_profiles variable to avoid mypy no-red…

2b0fe6b

…ef error Signed-off-by: Himanshu Singh <himanshu.singh@walmart.com>

fix: rename Branch A cluster_kwargs to avoid mypy no-redef with Branch B

5c4a712

Signed-off-by: Himanshu Singh <himanshu.singh@walmart.com>

fix: rename Branch A cluster_kwargs to avoid mypy no-redef with Branch B

5c17b8d

Signed-off-by: Himanshu Singh <himanshu.singh@walmart.com>

regenerated pixi

c42bb2b

Signed-off-by: Himanshu Singh <himanshu.singh@walmart.com>

trigger build

76535a5

Signed-off-by: Himanshu Singh <himanshu.singh@walmart.com>

singhhimanshu0811 force-pushed the feat/cassandra-multi-dc-execution-profiles branch from 4f7ea18 to 76535a5 Compare June 28, 2026 07:59

ntkathole approved these changes Jun 28, 2026

View reviewed changes

ntkathole merged commit 0de9196 into feast-dev:master Jun 28, 2026
24 checks passed

Navigation Menu

Uh oh!

feat(cassandra): Add multi-DC support via per-datacenter execution profiles - #6434

feat(cassandra): Add multi-DC support via per-datacenter execution profiles#6434
ntkathole merged 15 commits into
feast-dev:masterfrom
singhhimanshu0811:feat/cassandra-multi-dc-execution-profiles

singhhimanshu0811 commented May 24, 2026 •

edited

Loading

Uh oh!

ntkathole left a comment

Uh oh!

singhhimanshu0811 commented May 25, 2026

Uh oh!

singhhimanshu0811 commented May 25, 2026 •

edited

Loading

Uh oh!

ntkathole commented May 26, 2026

Uh oh!

ntkathole commented Jun 9, 2026

Uh oh!

franciscojavierarceo commented Jun 17, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

singhhimanshu0811 commented Jun 28, 2026 •

edited

Loading

Uh oh!

ntkathole commented Jun 28, 2026

Uh oh!

singhhimanshu0811 commented Jun 28, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

singhhimanshu0811 commented May 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Motivation

Backward compatibility

Uh oh!

ntkathole left a comment

Choose a reason for hiding this comment

Review of PR #6434 — Multi-DC Cassandra Support

1. Bug: Mutable class-level default for _dc_execution_profiles

2. Bug: Missing mutual exclusivity validation

3. Design: Profiles are created but never used by Feast operations

4. Design: replication_factor and replication_strategy are purely informational

5. Suggestion: Duplicate local_dc naming is confusing

6. Suggestion: Empty datacenters list edge case

7. Tests needed

8. Minor: Redundant if not self._session: in Branch A

Summary

Uh oh!

singhhimanshu0811 commented May 25, 2026

Uh oh!

singhhimanshu0811 commented May 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ntkathole commented May 26, 2026

Uh oh!

ntkathole commented Jun 9, 2026

Uh oh!

franciscojavierarceo commented Jun 17, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

singhhimanshu0811 commented Jun 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ntkathole commented Jun 28, 2026

Uh oh!

singhhimanshu0811 commented Jun 28, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

singhhimanshu0811 commented May 24, 2026 •

edited

Loading

1. Bug: Mutable class-level default for `_dc_execution_profiles`

4. Design: `replication_factor` and `replication_strategy` are purely informational

5. Suggestion: Duplicate `local_dc` naming is confusing

6. Suggestion: Empty `datacenters` list edge case

8. Minor: Redundant `if not self._session:` in Branch A

singhhimanshu0811 commented May 25, 2026 •

edited

Loading

singhhimanshu0811 commented Jun 28, 2026 •

edited

Loading