Skip to content

Data science onramp Data ingestion#4447

Merged
leahecole merged 105 commits intoGoogleCloudPlatform:masterfrom
leahecole:data-ingestion
Aug 14, 2020
Merged

Data science onramp Data ingestion#4447
leahecole merged 105 commits intoGoogleCloudPlatform:masterfrom
leahecole:data-ingestion

Conversation

@Symmetries
Copy link
Copy Markdown
Contributor

@Symmetries Symmetries commented Aug 6, 2020

Description

Fixes #

Note: It's a good idea to open an issue first for discussion.

Checklist

vuppalli and others added 30 commits June 5, 2020 15:35
@vuppalli
add data ingestion code
92cf763
@Symmetries
begin addressing comments
739114a
@vuppalli
change submit job
681eaf3
@Symmetries
address code structure and global variable issues
4afbf1c
@vuppalli
get dataproc job output and fix linting
744f80c
@vuppalli
fix PR comments
8cd7dc6
@vuppalli
linting and global vars
81265d2
@vuppalli
address Brad PR comments
3e86bda
@tk755
broken clean.py
580c8e1
@tk755
Revert "broken clean.py"
4ed5a15 
This reverts commit 580c8e1.
@Symmetries
optimize data ingestion
e6fe99d
@vuppalli
fix linting errors
540acaa
@Symmetries
fix minor style issues
a7e2972
@Symmetries
remove pip from cluster config
3e5ba3b
@Symmetries
load external datasets from url
2106153
@vuppalli
add citibike dataset notebook
0692122
@vuppalli
address leahs comments
01eb916
@vuppalli
add gas dataset code
56faae7
@vuppalli
add scikit learn
5c672c4
@vuppalli
rename file
f6faeeb
@vuppalli
small wording change
d47841c
@vuppalli
address brad and diego comments
d548dd0
@vuppalli
fix linting issues
129a5b9
@vuppalli
add outputs
5c284ea
@vuppalli
add US holidays feature engineering
6c565f4
@vuppalli
Delete noxfile.py
c9b5dd7
@vuppalli
minor changes
c5283c9
@vuppalli
change output
74881a9
@vuppalli
Merge branch 'feature-engineering' of https://github.com/Symmetries/p…
7d21865 
…ython-docs-samples into feature-engineering
@tk755
added dry-run flag
9febbad
bradmiro
bradmiro reviewed Aug 12, 2020
View reviewed changes
data-science-onramp/data-ingestion/setup.py Outdated
Comment on lines +22 to +23
BUCKET_NAME = sys.argv[1]
DATASET_NAME = sys.argv[2]
Copy link
Copy Markdown
Collaborator

@bradmiro bradmiro Aug 11, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move args under __main__

data-science-onramp/data-ingestion/setup.py Outdated
Comment on lines +87 to +88
if angle is None:
return None
Copy link
Copy Markdown
Collaborator

@bradmiro bradmiro Aug 11, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: if not angle:

data-science-onramp/data-ingestion/setup_test.py Outdated
"--test",
],
"jar_file_uris": [
"gs://spark-lib/bigquery/spark-bigquery-latest_2.12.jar"
Copy link
Copy Markdown
Collaborator

@bradmiro bradmiro Aug 11, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spacing is too large here. Probably needs to be linted.

data-science-onramp/data-ingestion/setup_test.py Outdated
Comment on lines +78 to +79
operation = cluster_client.create_cluster(PROJECT, REGION, CLUSTER_DATA)

Copy link
Copy Markdown
Collaborator

@bradmiro bradmiro Aug 11, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On Monday a new version of the python-dataproc library was released with breaking changes. project_id, region and cluster are now key word args. Can you please update accordingly? I'm happy to help with this also.

data-science-onramp/data-ingestion/setup_test.py Outdated
Comment on lines +86 to +87
operation = cluster_client.delete_cluster(PROJECT, REGION,
CLUSTER_NAME)
Copy link
Copy Markdown
Collaborator

@bradmiro bradmiro Aug 11, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change to project_id=PROJECT, region=REGION, cluster_name=CLUSTER_NAME please.

data-science-onramp/data-ingestion/setup_test.py Outdated
Comment on lines +147 to +148
response = job_client.submit_job_as_operation(project_id=PROJECT, region=REGION,
job=JOB_DETAILS)
Copy link
Copy Markdown
Collaborator

@bradmiro bradmiro Aug 11, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks correct with respect to API update.

bradmiro
bradmiro reviewed Aug 12, 2020
View reviewed changes
data-science-onramp/data-ingestion/requirements.txt Outdated
google-auth-httplib2==0.0.3
google-cloud==0.34.0
google-cloud-storage==1.28.1
google-cloud-dataproc==0.8.0
Copy link
Copy Markdown
Collaborator

@bradmiro bradmiro Aug 12, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

google-cloud-dataproc==2.0.0

@google-cla
Copy link
Copy Markdown

google-cla bot commented Aug 12, 2020

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

@google-cla
Copy link
Copy Markdown

google-cla bot commented Aug 13, 2020

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

@google-cla
Copy link
Copy Markdown

google-cla bot commented Aug 13, 2020

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

@leahecole leahecole added cla: yes This human has signed the Contributor License Agreement. and removed cla: no This human has *not* signed the Contributor License Agreement. labels Aug 13, 2020
@leahecole leahecole changed the title WIP: Data science onramp Data ingestion Data science onramp Data ingestion Aug 13, 2020
@google-cla
Copy link
Copy Markdown

google-cla bot commented Aug 13, 2020

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

@google-cla google-cla bot added cla: no This human has *not* signed the Contributor License Agreement. and removed cla: yes This human has signed the Contributor License Agreement. labels Aug 13, 2020
@google-cla
Copy link
Copy Markdown

google-cla bot commented Aug 13, 2020

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

@google-cla google-cla bot added cla: yes This human has signed the Contributor License Agreement. and removed cla: no This human has *not* signed the Contributor License Agreement. labels Aug 13, 2020
@leahecole leahecole added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 13, 2020
@leahecole
Copy link
Copy Markdown
Collaborator

leahecole commented Aug 13, 2020

gonna rerun to see if it's a flake and there were other tests running

@kokoro-team kokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 13, 2020
@leahecole leahecole merged commit ca4e6a4 into GoogleCloudPlatform:master Aug 14, 2020
@tmatsuo
Copy link
Copy Markdown
Contributor

tmatsuo commented Aug 14, 2020

@leahecole @bradmiro This PR deletes the top level .gitignore. Please add it back.

@tmatsuo
Copy link
Copy Markdown
Contributor

tmatsuo commented Aug 15, 2020

nm, #4502

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@leahecole leahecole leahecole approved these changes

@bradmiro bradmiro bradmiro approved these changes

Assignees

@leahecole leahecole

Labels

cla: yes This human has signed the Contributor License Agreement.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@Symmetries @leahecole @tmatsuo @bradmiro @kokoro-team @vuppalli @tk755