Skip to content

ppai/weather: add timeout to vertex training - #9273

Merged
davidcavazos merged 1 commit into
GoogleCloudPlatform:mainfrom
davidcavazos:ppai-weather-forecasting
Mar 13, 2023
Merged

ppai/weather: add timeout to vertex training#9273
davidcavazos merged 1 commit into
GoogleCloudPlatform:mainfrom
davidcavazos:ppai-weather-forecasting

Conversation

@davidcavazos

@davidcavazos davidcavazos commented Mar 13, 2023 •
edited
Loading

Copy link
Copy Markdown

Description

Adds a timeout to the Vertex AI training job to avoid having long running jobs during testing.
The training job should take a couple seconds to finish after setting up the VM, I added a 5 minutes timeout which should be more than enough for the job to finish.

⚠️ Note: Tests will still fail until #9272 is fixed, but this will avoid the long running hanged jobs.

Additional minor changes:

  • Update the prebuilt container version.
  • Include torch in the package requirements rather than in the testing requirements.

Checklist

@davidcavazos
davidcavazos requested a review from a team as a code owner March 13, 2023 22:06
@review-notebook-app

review-notebook-app Bot commented Mar 13, 2023

Copy link
Copy Markdown

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

@product-auto-label product-auto-label Bot added samples Issues that are directly related to samples. api: notebooks Issues related to the Agent Platform Workbench API. labels Mar 13, 2023
@davidcavazos

davidcavazos commented Mar 13, 2023

Copy link
Copy Markdown
Author

Confirmed, the training job timed out at 5 minutes, which is the expected error we wanted.

RuntimeError: Training failed with:
code: 1
message: "CANCELED"

@davidcavazos
davidcavazos merged commit 3851704 into GoogleCloudPlatform:main Mar 13, 2023
@davidcavazos
davidcavazos deleted the ppai-weather-forecasting branch March 13, 2023 22:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kweinmeister kweinmeister kweinmeister approved these changes
+1 more reviewer
@andrewferlitsch andrewferlitsch andrewferlitsch approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

@engelke engelke

Labels

api: notebooks Issues related to the Agent Platform Workbench API. samples Issues that are directly related to samples.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@davidcavazos @kweinmeister @andrewferlitsch @engelke