feat: Implement Databricks Unity Catalog offline store integration#6515
feat: Implement Databricks Unity Catalog offline store integration#6515falloficaruss wants to merge 3 commits into
Conversation
Signed-off-by: Abhishek Shinde <norizzabhii@gmail.com>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3a4edcdb4b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| return spark_session | ||
|
|
||
|
|
||
| class DatabricksUCOfflineStore(SparkOfflineStore): |
There was a problem hiding this comment.
Initialize Databricks sessions during source validation
When users run feast apply or feast plan with source validation enabled, this subclass still inherits OfflineStore.validate_data_source and get_table_column_names_and_types_from_data_source, so validation goes through SparkSource.validate() and starts a plain Spark session via get_spark_session_or_start_new_with_repoconfig instead of this Databricks helper. In a local Databricks Connect setup with no active Databricks session, UC tables are validated against a local/non-UC Spark session and fail before any of the wrapped retrieval methods run; override these validation/schema methods to initialize the Databricks session too.
Useful? React with 👍 / 👎.
…tion methods Signed-off-by: Abhishek Shinde <norizzabhii@gmail.com>
What this PR does / why we need it:
Which issue(s) this PR fixes:
Partially Fixes - #6499
Checks
git commit -s)Testing Strategy