Skip to content

Add Synapse Launcher #61

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
xiaoyongzhu wants to merge 23 commits into
base: master
Choose a base branch
from
Open

Add Synapse Launcher #61

Changes from 1 commit
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
8a1a310
Add Synapse Launcher
xiaoyongzhu Apr 25, 2021
8c06c45
Remove unnecessary printout
xiaoyongzhu Apr 26, 2021
b3fd983
Add eventhub support
xiaoyongzhu May 8, 2021
24f0986
Merge branch 'master' of https://github.com/xiaoyongzhu/feast-spark
xiaoyongzhu May 19, 2021
89f48a3
Add EventHub support and Redis Auth support
xiaoyongzhu May 19, 2021
f2cd8be
Adding EventHub support in Spark jobs
xiaoyongzhu May 19, 2021
53d7e20
add ScheduledBatchIngestionJobParameters
xiaoyongzhu Aug 31, 2021
79866e9
Merge pull request #1 from feast-dev/master
xiaoyongzhu Aug 31, 2021
c469ee7
Add Azure specific dependencies
xiaoyongzhu Sep 2, 2021
91c6822
Change azure storage dependencies
xiaoyongzhu Sep 2, 2021
0a7a56c
Commen for removing/adding spaces between brackets
xiaoyongzhu Sep 9, 2021
f4c0d5a
Delete feature_store_debug.py
xiaoyongzhumsft Sep 9, 2021
6bc9260
Update StreamingPipeline.scala
xiaoyongzhumsft Sep 14, 2021
dd53a53
Merge branch 'feast-dev:master' into master
xiaoyongzhumsft Oct 1, 2021
9fde235
Update synapse.py
xiaoyongzhumsft Oct 1, 2021
bb3d6be
Update synapse.py
xiaoyongzhumsft Oct 1, 2021
6885f31
Merge branch 'feast-dev:master' into master
xiaoyongzhumsft Oct 4, 2021
08da84f
Fix Redis auth issue
xiaoyongzhu Oct 12, 2021
0ddbcef
Update Ingestion jobs and add supporting files
xiaoyongzhu Oct 12, 2021
48a1c44
Fix build issues
xiaoyongzhu Oct 12, 2021
762386e
Add support for Kafka ingestion
xiaoyongzhu Oct 12, 2021
41fc406
Add build and push instructions
xiaoyongzhu Oct 12, 2021
0f7d433
Adding License
xiaoyongzhu Oct 27, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update synapse.py 
Change the authentication chain (first using Environmental variables then using device login)
  • Loading branch information
@xiaoyongzhumsft
xiaoyongzhumsft authored Oct 1, 2021
commit bb3d6be54f6e42f45925696db662fb29ca297e13
10 changes: 3 additions & 7 deletions python/feast_spark/pyspark/launchers/synapse/synapse.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -160,19 +160,15 @@ def __init__(
global login_credential_cache
# use a global cache to store the credential, to avoid users from multiple login

# if self.credential is None:
# prioritize using service principal
self.credential = DefaultAzureCredential()
login_credential_cache = self.credential
if login_credential_cache is None:
# use DeviceCodeCredential if DefaultAzureCredential is not available
self.credential = DeviceCodeCredential(client_id, authority=authority_host_uri, tenant=tenant_id)
# use DeviceCodeCredential if EnvironmentCredential is not available
self.credential = ChainedTokenCredential(EnvironmentCredential(), DeviceCodeCredential(client_id, authority=authority_host_uri, tenant=tenant_id))
login_credential_cache = self.credential
else:
self.credential = login_credential_cache

self._api = SynapseJobRunner(synapse_dev_url, pool_name, executor_size = executor_size, executors = executors, credential=self.credential)
self._datalake = DataLakeFiler(datalake_dir)
self._datalake = DataLakeFiler(datalake_dir,credential=self.credential)

def _job_from_job_info(self, job_info: SparkBatchJob) -> SparkJob:
job_type = job_info.tags[LABEL_JOBTYPE]
Expand Down