Skip to content

Add [all] extras to install all extra dependencies #7610

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tswast merged 4 commits into from
Apr 9, 2019
Merged

Add [all] extras to install all extra dependencies #7610

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
85a34ca
Add [all] extras to install all extra dependencies
tswast Mar 29, 2019
8faed78
Need fastavro to actually read rows from bqstorage.
tswast Mar 30, 2019
aec0163
Blacken
tswast Apr 1, 2019
17e65d7
Use lists for all extras.
tswast Apr 1, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 5 additions & 14 deletions bigquery/noxfile.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,13 +20,7 @@
import nox


LOCAL_DEPS = (
os.path.join("..", "api_core[grpc]"),
os.path.join("..", "core"),
# TODO: Move bigquery_storage back to dev_install once dtypes feature is
# released. Issue #7049
os.path.join("..", "bigquery_storage[pandas,fastavro]"),
)
LOCAL_DEPS = (os.path.join("..", "api_core[grpc]"), os.path.join("..", "core"))

BLACK_PATHS = ("google", "tests", "docs", "noxfile.py", "setup.py")

Expand All @@ -45,10 +39,7 @@ def default(session):
session.install("-e", local_dep)

# Pyarrow does not support Python 3.7
if session.python == "3.7":
dev_install = ".[pandas, tqdm]"
else:
dev_install = ".[pandas, pyarrow, tqdm]"
dev_install = ".[all]"
session.install("-e", dev_install)

# IPython does not support Python 2 after version 5.x
Expand Down Expand Up @@ -95,7 +86,7 @@ def system(session):
session.install("-e", local_dep)
session.install("-e", os.path.join("..", "storage"))
session.install("-e", os.path.join("..", "test_utils"))
session.install("-e", ".[pandas]")
session.install("-e", ".[all]")

# IPython does not support Python 2 after version 5.x
if session.python == "2.7":
Expand Down Expand Up @@ -123,7 +114,7 @@ def snippets(session):
session.install("-e", local_dep)
session.install("-e", os.path.join("..", "storage"))
session.install("-e", os.path.join("..", "test_utils"))
session.install("-e", ".[pandas, pyarrow, fastparquet]")
session.install("-e", ".[all]")

# Run py.test against the snippets tests.
session.run("py.test", os.path.join("docs", "snippets.py"), *session.posargs)
Expand Down Expand Up @@ -182,7 +173,7 @@ def docs(session):
for local_dep in LOCAL_DEPS:
session.install("-e", local_dep)
session.install("-e", os.path.join("..", "storage"))
session.install("-e", ".[pandas, pyarrow]")
session.install("-e", ".[all]")

shutil.rmtree(os.path.join("docs", "_build"), ignore_errors=True)
session.run(
Expand Down
19 changes: 15 additions & 4 deletions bigquery/setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,14 +34,25 @@
"google-resumable-media >= 0.3.1",
]
extras = {
"bqstorage": "google-cloud-bigquery-storage >= 0.2.0dev1, <2.0.0dev",
"pandas": "pandas>=0.17.1",
"bqstorage": [
"google-cloud-bigquery-storage >= 0.2.0dev1, <2.0.0dev",
"fastavro>=0.21.2",
],
"pandas": ["pandas>=0.17.1"],
# Exclude PyArrow dependency from Windows Python 2.7.
'pyarrow: platform_system != "Windows" or python_version >= "3.4"': "pyarrow>=0.4.1",
"tqdm": "tqdm >= 4.0.0, <5.0.0dev",
'pyarrow: platform_system != "Windows" or python_version >= "3.4"': [
"pyarrow>=0.4.1"
],
"tqdm": ["tqdm >= 4.0.0, <5.0.0dev"],
"fastparquet": ["fastparquet", "python-snappy"],
}

all_extras = []
Comment thread
tswast marked this conversation as resolved.

for extra in extras:
all_extras.extend(extras[extra])

extras["all"] = all_extras

@tseaver tseaver Mar 29, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't we just normalize the extras above to always be lists?

@tswast tswast Mar 29, 2019

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I fear if we miss one, then we end up with a bad dependency list.

>>> l = []
>>> l.extend("Uh, oh!")
>>> l
['U', 'h', ',', ' ', 'o', 'h', '!']

That means we still would need a check for string type or something else to validate they are all lists.

@tswast tswast Mar 30, 2019

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't what I was thinking with that comment. This is why we have tests. I assume by using the [all] extra in the nox session, we'd catch such errors. I'll simplify this on Monday as you suggest.

@tswast tswast Apr 1, 2019

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 17e65d7.


# Setup boilerplate below this line.

Expand Down