-
Notifications
You must be signed in to change notification settings - Fork 1.3k
feat: Contrib azure provider with synapse/mssql offline store and Azure registry store #3072
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 1 commit
e1e210d
011d1e0
a6a2fce
57b63bb
ae7ed8a
421645b
07fece5
4062031
cb39329
554ca1a
4a969e7
116320a
c0b16ef
44d09d0
b6f0a79
2b2ff40
4616366
c7d9852
d2e290b
a726a9a
32992e3
ebb934b
e456acb
45f479f
4b8c4a2
b1bf602
4586f00
3b88c0b
9ae8ee3
1b12e4a
0ca5048
883f314
ccf8716
f05288e
ee30e73
ab17db9
be162f5
f5aa476
4423dfa
5806507
7a4d055
78b74b1
a9e8119
1341e3e
3d42093
1c591f0
b4da607
c3a0423
576b57e
69940ac
516ff76
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
Signed-off-by: Kevin Zhang <kzhang@tecton.ai>
- Loading branch information
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,6 +1,6 @@ | ||
| # Copyright (c) Microsoft Corporation. | ||
| # Licensed under the MIT license. | ||
|
|
||
| import warnings | ||
| from datetime import datetime, timedelta | ||
| from typing import Any, Callable, Dict, List, Optional, Sequence, Tuple, Union | ||
|
|
||
|
|
@@ -28,11 +28,19 @@ | |
| make_tzaware, | ||
| ) | ||
|
|
||
| # Make sure spark warning doesn't raise more than once. | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. not spark
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. as discussed, we can kill this provider probably or use passthroughprovider
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. removed |
||
| warnings.simplefilter("once", RuntimeWarning) | ||
|
|
||
| DEFAULT_BATCH_SIZE = 10_000 | ||
|
|
||
|
|
||
| class AzureProvider(Provider): | ||
| def __init__(self, config: RepoConfig): | ||
| warnings.warn( | ||
| "The azure provider is an experimental feature in alpha development. " | ||
| "Some functionality may still be unstable so functionality can change in the future.", | ||
| RuntimeWarning, | ||
| ) | ||
| self.repo_config = config | ||
| self.offline_store = get_offline_store_from_config(config.offline_store) | ||
| self.online_store = ( | ||
|
|
@@ -69,6 +77,11 @@ def teardown_infra( | |
| tables: Sequence[FeatureView], | ||
| entities: Sequence[Entity], | ||
| ) -> None: | ||
| warnings.warn( | ||
| "The azure provider is an experimental feature in alpha development. " | ||
| "Some functionality may still be unstable so functionality can change in the future.", | ||
| RuntimeWarning, | ||
| ) | ||
| if self.online_store: | ||
| self.online_store.teardown(self.repo_config, tables, entities) | ||
|
|
||
|
|
@@ -93,6 +106,7 @@ def online_read( | |
| entity_keys: List[EntityKeyProto], | ||
| requested_features: List[str] = None, | ||
| ) -> List[Tuple[Optional[datetime], Optional[Dict[str, ValueProto]]]]: | ||
|
|
||
| result = [] | ||
| if self.online_store: | ||
| result = self.online_store.online_read( | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should this be Azure Synapse + Azure SQL?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed