chore: merge dev into main for MB SDK Preview#305
Merged
Merged
Conversation
…er (#18) * Stub out initial MB classes(b/169782786) and add naive implementation of initializer(b/169762813). * add license * Add license to missed models.py and fixed spacing * Move modules to to aiplatform level
* Add initial helper util functions and tests * Make requested changes, add TODOs to utils.py * Fix indent and lint
#26) * Add support for default project, credentials, and location to initalizer. Initializer tests. Moved initalizer based utils from utils to Init class. Formatting fixes.
* Add initial implementation of Model class along with AI Platform resource noun base class.
* Export aiplatform_v1beta1 as aiplatform.gapic_v1beta1 (#17) Freeing up the google.cloud.aiplatform namespace for MB SDK * GAPIC namespace (#20) * name space google.cloud.aiplatform.gapic * Init Disclaimer (#28) * clean up docstrings that are not rendered properly in reference doc (#24) Co-authored-by: Vinny Senthil <vinnysenthil@gmail.com> * fix: re-add py sessions to noxfile (#22) * fix: re-add py sessions to noxfile Co-authored-by: Vinny Senthil <vinnysenthil@gmail.com> Co-authored-by: Yu-Han Liu <yuhanliu@google.com> Co-authored-by: Vinny Senthil <vinnysenthil@gmail.com>
* Dataset class (#1) * Add global_config unset project error * Add two validation functions to utils * Implement initial Dataset class * Lint utils.py * Address reviewer comments + remove aip alias * Change re.Match to typing.Match * Lint with Py 3.8 * Address flake8 errors, remove unused vars
…34) Initial implementation of python script packager.
* first lro attempt * address comments * add test Co-authored-by: Yu-Han Liu <yuhanliu@google.com>
…ion (#40) * feat: Add tabular import support from GCS and BQ * fix: Ran linter again with following commands pip3 install flake8 black==19.10b0 black docs google tests noxfile.py setup.py flake8 google tests * fix: Moved instantiation after validation and renamed a _import_gcs function Co-authored-by: Ivan Cheung <ivanmkc@google.com>
* feat: Add Custom Training by Python Script * chore: fix LRO tests and formatting * feat: Address comments and change args to accept list of args to support positional args and flags. * feat: rename is_failed to has_failed * refactor: remove unused Dict type * chore: lint
* recs * create endpoint * deploy model * undeploy model * predict placeholders * use base properties * address feedback * address feedback * use full name * address comments * feat: Add Custom Training by Python Script (#6) * chore: fix LRO tests and formatting * add tests * fix typo * more typos * address feedback * feat: Address comments and change args to accept list of args to support positional args and flags. * feat: rename is_failed to has_failed * refactor: remove unused Dict type * Work on predict (#7) * fix tests * fix tests * fix tests * checkpoint * Endpoint (#8) * recs * create endpoint * deploy model * undeploy model * predict placeholders * use base properties * address feedback * address feedback * use full name * address comments * add tests * fix typo * more typos * address feedback * fix tests * fix tests * fix tests Co-authored-by: sirtorry <torryyang@google.com> Co-authored-by: Torry Yang <sirtorry@users.noreply.github.com> * address feedback * add tests * appease linters * fix test * fix test * add self to test params * tests * tests * tests * fix tests * more bug crushing * undo 'fix' * test: fix endpoint test * chore: change explain ticket ref * test: fix endpoint tests Co-authored-by: sasha-gitg <44654632+sasha-gitg@users.noreply.github.com> Co-authored-by: Sasha Sobran <asobran@google.com>
* env var helpers
* check display name len
… training jobs (#49) * Extracted reusable CustomTrainingJob code into TrainingJob base class Moved around some functions Completed refactor to use base class bug: remove requirement for import_schema_uri when passing in gcs_source (#46) Ran linters Removed model from TrainingJob and moved to CustomTrainingJob Removed DatasetWithSplits Added doc strings and simplified training_job_base code Moved TrainingJob class into training_jobs.py Removed container_uri from base TrainingJob class Addressed comments Fixed managed model Ran linter Fixed issues with abc, doc string and super call Refactored to create input data config separately * Ran linter Co-authored-by: Ivan Cheung <ivanmkc@google.com>
* Create and move all constants to constants.py * Fix tests after constants.py, drop unused vars * Init Job and BatchPredictionJob class, unit tests * Address all reviewer comments * Update docstring to bigquery.table.RowIterator * Get GCS/BQ clients to use same creds as uCAIP
Added the training job subclass for tables Co-authored-by: Ivan Cheung <ivanmkc@google.com>
Co-authored-by: Ivan Cheung <ivanmkc@google.com>
Co-authored-by: Ivan Cheung <ivanmkc@google.com>
* feat: Added predefined splits to custom training jobs * Added tests * Added docstrings, fixed run functions and fixed unit test * Added support for predefined splits in AutoMLTabularTrainingJob * Consolidated _create_input_data_config and fixed linting * Switched from gcs_destination to gcs_destination_uri_prefix Co-authored-by: Ivan Cheung <ivanmkc@google.com>
…splits (#99) * fix: Added check for tabular dataset when using pre-defined splits * Ran linters * fix: Small cleanup of control flow Co-authored-by: Ivan Cheung <ivanmkc@google.com>
* Fix gs:// prefix bug in BatchPredictionJob class * Implement Model.batch_predict() * Add unit tests for Model.batch_predict() * Remove location param from batch_predict() * Use direct module import, fix doc strings * Lint test_jobs.py, models.py * Add util helper to split gcs_path
Co-authored-by: Ivan Cheung <ivanmkc@google.com>
…nchronous (#291) * feature: Use gca_resource as repr and provide informative repr if asynchronous * feature: change repr from gca resource to resource name
…m compat.V1BETA1 to compat.V1 (#290) * resolving rebase conflict, review feedback * change compat.DEFAULT_VERSION to compat.V1; change tests services and types to V1 for general; add fixtures to support V1BETA1 tests for explanation * chore(deps): update dependency google-cloud-aiplatform to v0.6.0 (#270) * fix: add v1 conversion value rule Co-authored-by: WhiteSource Renovate <bot@renovateapp.com> Co-authored-by: Sasha Sobran <asobran@google.com>
…CI (#279) These file additions allow for the `samples/model-builder/*` sample tests to be picked up and run by Kokoro CI which currently only runs tests in `samples/snippets/*`
### [Colab for manual testing](https://colab.research.google.com/drive/14iLNaJEyZaPGebCgJZxgUJJeU9r7z_Fu) ### Summary of Changes - Added a `_list_method` property to `AiPlatformResourceNoun` to store GAPIC method name for each noun - Added a `create_time` and `update_time` property to `AiPlatformResourceNoun` - Added a single `list()` method that takes four optional fields and returns a list of SDK types - All of the fields except `order_by` are available in every GAPIC list methods - Added local sorting for GAPIC list methods that do not take `order_by` - Added 3 unit tests to check correct GAPIC calls and local sorting - Added `aiplatform.init()` to test class setup and dropped it from some unit tests Fixes [b/183498826](http://b/183498826) 🦕
|
Here is the summary of changes. You are about to add 11 region tags.
This comment is generated by snippet-bot.
|
- Removes all use of `aiplatform.Dataset` - Replaces internal references with `aiplatform._Dataset` - Replaces public references with specific dataset sub-class or TypeVar `datasets.SupportedDatasets` ## TODO after merging - [x] @sasha-gitg PTAL [at this list of updates](https://docs.google.com/document/d/1mKqldyCZaLgtYrgxpd6F3TJ35Ap3l6MhOxyiWTmZ4NQ/edit) to external notebooks and Colabs to no longer reference `aiplatform.Dataset` Fixes [b/184156060](http://b/184156060) 🦕
chore: merge main into dev
* samples: NLP import and predict samples for SDK * fix: mismatched region tags * samples: add more import data samples * samples: adds prediction NLP samples * fix: lint * samples: prediction tests * samples: adds endpoint fixtures * fix: lint * Update all MBSDK Dataset sample test mocks, fix NLP sample tests * Remove reference to ipdb Co-authored-by: Vinny Senthil <vinnysenthil@gmail.com>
vinnysenthil
approved these changes
Apr 13, 2021
dizcology
approved these changes
Apr 13, 2021
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This should be good to go pending this PR: #306.