Skip to content

Allow users compute statistics over retrieved batch datasets - #799

Merged
feast-ci-bot merged 6 commits into
feast-dev:masterfrom
zhilingc:retrieval-stats
Jun 23, 2020
Merged

Allow users compute statistics over retrieved batch datasets#799
feast-ci-bot merged 6 commits into
feast-dev:masterfrom
zhilingc:retrieval-stats

Conversation

@zhilingc

@zhilingc zhilingc commented Jun 16, 2020

Copy link
Copy Markdown
Collaborator

What this PR does / why we need it:
This PR adds the functionality to allow users to optionally compute tfdv statistics over retrieved batch datasets:

# compute_statistics defaults to false. If false, statistics() will always return an empty object.
dataset = client.get_batch_features(feature_refs, entity_rows, compute_statistics=True)
# waits for retrieval job to be complete, then returns tfdv statistics
stats = dataset.statistics()

Like with the feature set statistics, these statstics can then be fed into tfdv for validation, or facets to explore the dataset.

Which issue(s) this PR fixes:

Fixes #

Does this PR introduce a user-facing change?:

- Adds the parameter compute_statistics to client.get_batch_statistics()
- Adds the method statistics() which returns statistics from retrieval job

woop
woop reviewed Jun 17, 2020
View reviewed changes
Comment thread protos/feast/serving/ServingService.proto Outdated
@zhilingc

zhilingc commented Jun 17, 2020

Copy link
Copy Markdown
Collaborator Author

/test test-end-to-end-batch

woop
woop reviewed Jun 17, 2020
View reviewed changes
Comment thread tests/e2e/bq/bq-batch-retrieval.py Outdated
@zhilingc

zhilingc commented Jun 18, 2020

Copy link
Copy Markdown
Collaborator Author

/test test-end-to-end-batch

1 similar comment
@zhilingc

zhilingc commented Jun 18, 2020

Copy link
Copy Markdown
Collaborator Author

/test test-end-to-end-batch

@zhilingc
zhilingc force-pushed the retrieval-stats branch 2 times, most recently from f404741 to e8ae893 Compare June 23, 2020 03:46
@zhilingc

zhilingc commented Jun 23, 2020

Copy link
Copy Markdown
Collaborator Author

/test test-end-to-end-batch

@zhilingc

zhilingc commented Jun 23, 2020

Copy link
Copy Markdown
Collaborator Author

/test test-end-to-end-redis-cluster

@woop

woop commented Jun 23, 2020

Copy link
Copy Markdown
Member

/lgtm

@feast-ci-bot

feast-ci-bot commented Jun 23, 2020

Copy link
Copy Markdown
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: woop, zhilingc

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@woop

woop commented Jun 23, 2020

Copy link
Copy Markdown
Member

/kind feature

@feast-ci-bot feast-ci-bot added kind/feature New feature or request and removed needs-kind labels Jun 23, 2020
@feast-ci-bot
feast-ci-bot merged commit 8c2201c into feast-dev:master Jun 23, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@woop woop woop approved these changes
@davidheryanto davidheryanto Awaiting requested review from davidheryanto
@khorshuheng khorshuheng Awaiting requested review from khorshuheng
@pyalex pyalex Awaiting requested review from pyalex

Assignees

@woop woop

Labels

approved kind/feature New feature or request lgtm size/XXL

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@zhilingc @woop @feast-ci-bot