Skip to content

Validating historical features against reference dataset with "great expectations" profiler #2243

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
feast-ci-bot merged 17 commits into from
Feb 2, 2022
Merged

Validating historical features against reference dataset with "great expectations" profiler #2243

Changes from 1 commit
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
038fd04
Validating historical features against reference dataset
pyalex Jan 6, 2022
3abda73
fixes after rebase
pyalex Jan 27, 2022
d4cbe97
fixes after rebase
pyalex Jan 27, 2022
3d7fcea
update ci requirements
pyalex Jan 27, 2022
07ae012
some reverts
pyalex Jan 27, 2022
2121ae5
format
pyalex Jan 27, 2022
af2576e
move ValidationReference proto message
pyalex Jan 27, 2022
17c4ec0
grammar in docs
pyalex Jan 27, 2022
d3449db
Update docs/reference/dqm.md
pyalex Feb 1, 2022
1ca7eaa
rebase & resolve conflicts
pyalex Feb 1, 2022
e377431
remove RetrievalJobWithValidation
pyalex Feb 1, 2022
0f87c7b
Update docs/reference/dqm.md
pyalex Feb 1, 2022
7c8660b
fix type hint
pyalex Feb 1, 2022
80f0814
fix function flow
pyalex Feb 1, 2022
3413587
more docstrings
pyalex Feb 1, 2022
059444e
update docs
pyalex Feb 1, 2022
c8d57c9
improve docs
pyalex Feb 1, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update docs/reference/dqm.md 
Co-authored-by: Danny Chiao <d.chiao@gmail.com>
Signed-off-by: pyalex <moskalenko.alexey@gmail.com>
  • Loading branch information
@pyalex @adchia
pyalex and adchia committed Feb 1, 2022
commit d3449db3f41d50c6ecb7a0e2bc7e932b9f404380
8 changes: 4 additions & 4 deletions docs/reference/dqm.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,13 @@ Validation could be applied during:
* [planned] Reading features from an online store

Its goal is to address several complex data problems, namely:
* Data Consistency - new training dataset could be significantly different from the previous, which might require a change in model architecture.
* Issues/bugs in the upstream pipeline - bug in upstream could cause invalid values to overwrite existing valid values in an online store.
* Data consistency - new training datasets can be significantly different from previous datasets. This might require a change in model architecture.
* Issues/bugs in the upstream pipeline - bugs in upstream pipelines can cause invalid values to overwrite existing valid values in an online store.
* Training/serving skew - distribution shift could significantly decrease the performance of the model.

> By “monitoring data quality” we understand verifying that the characteristics of the tested dataset (we call this characteristics dataset's profile) are "equivalent" to the characteristics of the reference dataset.
> To monitor data quality, we check that the characteristics of the tested dataset (aka the tested dataset's profile) are "equivalent" to the characteristics of the reference dataset.
> Eg, data currently passed to the model hasn’t changed significantly since the model was trained, and expectations implicitly made by ML algorithm during training are still met.
> How exactly profiles equivalency should be measured is up to the user.
> How exactly profile equivalency should be measured is up to the user.

### Overview

Expand Down