Skip to content

feat: MongoDB offline store #6138

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ntkathole merged 78 commits into from
Apr 30, 2026
Merged

feat: MongoDB offline store #6138

Changes from 1 commit
Commits
Show all changes
78 commits
Select commit Hold shift + click to select a range
d2c637d
feat: Add MongoDB offline store (ibis-based PIT join, v1 alpha)
caseyclements Mar 3, 2026
77f243e
refactor: improve MongoDB offline store code quality
caseyclements Mar 4, 2026
bb134a2
Started work on full Mongo/MQL implementation. Kept MongoDBOfflineSto…
caseyclements Mar 9, 2026
46e67b7
refactor: rename alpha to preview, clarify MQL pipeline comments
caseyclements Mar 17, 2026
d0c00ca
Added unit tests for offline store retrieval, requiring docker and py…
caseyclements Mar 17, 2026
c940d05
Added test of multiple feature views and compound join keys
caseyclements Mar 17, 2026
37fad38
Initial implementation of native single-collection offline store
caseyclements Mar 17, 2026
67c39d4
Added DriverInfo to MongoDBClients
caseyclements Mar 18, 2026
7987405
Optimized MQL. Applied FV-level TTL
caseyclements Mar 18, 2026
adf1fb0
filter TTL by relevant FVs only, cautiously reset df index; add creat…
caseyclements Mar 18, 2026
ed1571e
Updated docstrings
caseyclements Mar 18, 2026
c09faee
Lazy index creation via _get_client_and_ensure_indexes
caseyclements Mar 18, 2026
91e939c
Add performance benchmarks comparing Ibis vs Native MongoDB offline s…
caseyclements Mar 18, 2026
e4bfc31
Refactor Native get_historical_features: replace with fetch+pandas join
caseyclements Mar 18, 2026
6218fa8
Refactor get_historical_features with chunked processing for large en…
caseyclements Mar 19, 2026
b7ffd84
Optimize Native get_historical_features: reuse client, increase batch…
caseyclements Mar 19, 2026
affdc2d
Remove duplicate MongoDBOfflineStoreNative from mongodb.py
caseyclements Mar 19, 2026
afef4fd
Consolidate mongodb_source.py into mongodb.py
caseyclements Mar 19, 2026
f273744
Rename mongodb_offline_store to mongodb, use One/Many naming convention
caseyclements Mar 19, 2026
df00e47
Add README.md documenting MongoDB offline store implementations
caseyclements Mar 20, 2026
83d063b
Rename mongodb/ to mongodb_offline_store/, organize tests
caseyclements Mar 20, 2026
977c240
Update docstring in benchmark.py
caseyclements Mar 20, 2026
166d151
Update README to show created_at tie-breaker in Many schema
caseyclements Mar 20, 2026
fe24361
Update README index recommendations for Many implementation
caseyclements Mar 20, 2026
398110c
Add auto-create index to MongoDBOfflineStoreMany
caseyclements Mar 20, 2026
2feb97d
Update benchmark.py to use One/Many naming convention
caseyclements Mar 20, 2026
9624311
Add comprehensive module docstring to mongodb_many.py
caseyclements Mar 20, 2026
5c93a51
Add Feature Freshness and Schema Evolution docs to mongodb_many.py
caseyclements Mar 20, 2026
fd70c13
Add MongoDB DataSourceCreators for universal Feast tests
caseyclements Mar 20, 2026
b4a0260
Add .secrets.baseline
caseyclements Mar 20, 2026
9efd700
Addressed PR comment: join_keys = get_expected_join_keys(project, fea…
caseyclements Apr 14, 2026
5d67b3f
Adds tests scenario that not using offline_utils.get_expected_join_ke…
caseyclements Apr 14, 2026
f889e10
Tests revealed possible name collision in pandas.merge_asof
caseyclements Apr 15, 2026
fcbd609
Add further (Large) benchmark tests
caseyclements Apr 15, 2026
7786393
Upgrades from Devin comments. Class cache _index_initialized; get_exp…
caseyclements Apr 15, 2026
b8fa01c
Addressed PR comments
caseyclements Apr 15, 2026
6db7cce
Apply lower bound via max(TTL) when all feature viewws in a chunk hav…
caseyclements Apr 15, 2026
596b126
Add created_at to compound index so that materialization is correct i…
caseyclements Apr 15, 2026
c0173e0
Handdle numpy scalers in _serialize_entity_key_from_row as suggested.
caseyclements Apr 15, 2026
1f80f47
Add persist and tests
caseyclements Apr 15, 2026
daaaf32
Remove accidentally included design notes.
caseyclements Apr 15, 2026
3a78f9d
Fix entity key serialization: per-FV join key types and numpy 2.0 compat
caseyclements Apr 16, 2026
44abd92
Add offline_write_batch to MongoDBOfflineStoreOne
caseyclements Apr 16, 2026
63fab1b
mongodb_one: clarify pipeline sort rationale and avoid sparse-column …
caseyclements Apr 16, 2026
90dd224
Add mongodb_native.py: initial MQL-based offline store (pre-refactor …
caseyclements Apr 17, 2026
0733688
Refactor mongodb_native: Atlas-first $documents+$lookup PIT join
caseyclements Apr 17, 2026
47f1040
Add unit tests for MongoDBOfflineStoreNative
caseyclements Apr 17, 2026
84ac27b
Add cross-implementation equivalence suite (test_cross.py)
caseyclements Apr 17, 2026
8346f28
Add benchmark_sweep.py: four-dimensional scaling suite across all thr…
caseyclements Apr 17, 2026
448a698
Add mongodb_agg offline store — $match+$sort+$group, O(log P) without…
caseyclements Apr 19, 2026
810b7d0
Vectorize agg scoring path, add upfront index build, ignore design/
caseyclements Apr 19, 2026
2a56c13
Adds offline_write_batch
caseyclements Apr 20, 2026
7e35e6a
Adds detail to handling of K in benchmarks.
caseyclements Apr 20, 2026
fcbb8e1
Adds missing typing.
caseyclements Apr 21, 2026
6e8f502
Consolidate MongoDB offline store to single implementation
caseyclements Apr 23, 2026
a6dca86
Fixes strict_pit_false unit test.
caseyclements Apr 23, 2026
055d238
Fix MongoDB offline store: projection keying, TTL bounds, field mappi…
caseyclements Apr 24, 2026
136dc09
Fix MongoDB test DataSourceCreator: implement create_logged_features_…
caseyclements Apr 24, 2026
f7e6230
Fix pd.isna() ValueError on list/array features in offline_write_batch
caseyclements Apr 24, 2026
9d9f7f1
Merge branch 'master' into FEAST-OfflineStore-INTPYTHON-297
caseyclements Apr 24, 2026
274e6f8
Fix bool/int type inference order in get_table_column_names_and_types
caseyclements Apr 24, 2026
fcb92da
Fix mongodb_to_feast_value_type to accept type strings from get_table…
caseyclements Apr 24, 2026
23b6788
Sort join keys in _serialize_entity_key_from_row for consistent entit…
caseyclements Apr 24, 2026
26b5153
Resolve .secrets.baseline merge conflict with master
caseyclements Apr 24, 2026
5227804
Add mongodb to CI extras so pymongo is installed in CI
caseyclements Apr 24, 2026
2905b80
Remove MongoDB from universal test parametrization
caseyclements Apr 24, 2026
a9b9e8f
Regenerate pixi lockfile after pymongo addition to ci extras
caseyclements Apr 24, 2026
f558aa5
Fix scoring_path heuristic: check entity uniqueness per-FV, not globally
caseyclements Apr 28, 2026
349c5f1
Fix offline_write_batch: use original join key names for entity seria…
caseyclements Apr 28, 2026
8f8de11
Fix pull_latest and pull_all to return join key columns
caseyclements Apr 28, 2026
9de5548
Fix scoring_path: require homogeneous timestamps to prevent data loss
caseyclements Apr 29, 2026
c34b6cd
Fix training path: sort fv_df by created_at to break event_timestamp …
caseyclements Apr 29, 2026
1d2a1c9
Clean up stale docstrings: remove references to MongoDBOfflineStoreOn…
caseyclements Apr 29, 2026
02e457c
Clean up stale docstrings: remove references to MongoDBOfflineStoreOn…
caseyclements Apr 29, 2026
ae5256f
Added driver metadata to clients
caseyclements Apr 30, 2026
f1996fc
Update .secrets.baseline
caseyclements Apr 30, 2026
b3b7563
Remove preview warnings from MongoDB offline store
caseyclements Apr 30, 2026
4a43815
Merge branch 'master' into FEAST-OfflineStore-INTPYTHON-297
caseyclements Apr 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add .secrets.baseline 
Signed-off-by: Casey Clements <casey.clements@mongodb.com>
  • Loading branch information
@caseyclements
caseyclements committed Apr 24, 2026
commit b4a0260994014828aa10b81a012edd8c2680da7d
6 changes: 3 additions & 3 deletions .secrets.baseline
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1460,14 +1460,14 @@
"filename": "sdk/python/tests/universal/feature_repos/repo_configuration.py",
"hashed_secret": "d90e76ef629fb00c95f4e84fec29fbda111e2392",
"is_verified": false,
"line_number": 479
"line_number": 486
},
{
"type": "Secret Keyword",
"filename": "sdk/python/tests/universal/feature_repos/repo_configuration.py",
"hashed_secret": "5baa61e4c9b93f3f0682250b6cf8331b7ee68fd8",
"is_verified": false,
"line_number": 481
"line_number": 488
}
],
"sdk/python/tests/universal/feature_repos/universal/data_sources/file.py": [
Expand Down Expand Up @@ -1539,5 +1539,5 @@
}
]
},
"generated_at": "2026-03-20T20:27:19Z"
"generated_at": "2026-03-20T20:55:36Z"
}