Repository navigation
MatchingEngineIndex.create_tree_ah_index in Vertex Pipelines times out after 900 seconds #1870
Copy link
Copy link
Closed
Labels
api: vertex-aiIssues related to the googleapis/python-aiplatform API.Issues related to the googleapis/python-aiplatform API.
Description
Environment details
- Running in Vertex Pipelines
- OS type and version: gcr.io/deeplearning-platform-release/base-cu100
google-cloud-aiplatformversion: 1.20.0- KFP version: 1.8.18
Steps to reproduce
Build a kfp component using aiplatform.MatchingEngineIndex.create_tree_ah_index.
Include google-cloud-aiplatform==1.20.0 as a package to install. Use the gcr.io/deeplearning-platform-release/base-cu100 as the docker image. Build the pipeline json using kfp.v2.dsl's component function.
Create a Vertex Pipeline using the pipeline json.
Expected result
Pipeline should continue running until matching engine index is fully created.
Code Example
def create_tree_ah_index(
display_name: str,
jsonl_formatted_data_uri: str,
dimensions: int = 100,
approximate_neighbors_count: int = 150,
distance_measure_type: str = "DOT_PRODUCT_DISTANCE",
leaf_node_embedding_count: int = 500,
leaf_nodes_to_search_percent: float = 7,
description: str = "ANN index",
labels: dict = {"label_name": "label_value"},
sync: bool = False
):
from google.cloud import aiplatform
import logging
import time
logging.basicConfig(level=logging.INFO)
logging.info("ANN Index")
try:
tree_ah_index = aiplatform.MatchingEngineIndex.create_tree_ah_index(
display_name=display_name,
contents_delta_uri=jsonl_formatted_data_uri,
dimensions=dimensions,
approximate_neighbors_count=approximate_neighbors_count,
distance_measure_type=distance_measure_type,
leaf_node_embedding_count=leaf_node_embedding_count,
leaf_nodes_to_search_percent=leaf_nodes_to_search_percent,
description=description,
labels=labels,
sync=sync
)
while True:
if tree_ah_index._are_futures_done():
index_resource_name = tree_ah_index.resource_name
logging.info("Index successfully created with ID : %s ", index_resource_name)
break
logging.info("Polling the operation every 3 minutes to create index...")
time.sleep(180)
except Exception as e:
logging.exception("The index creation failed: {}".format(e))Stack trace
[KFP Executor 2022-12-22 18:33:31,182 INFO]: Create MatchingEngineIndex backing LRO: projects/589820861215/locations/us-central1/indexes/8058285535598215168/operations/2805451540867842048
[KFP Executor 2022-12-22 18:36:30,704 INFO]: Polling the operation every 3 minutes to create index...
[KFP Executor 2022-12-22 18:36:30,704 INFO]: Polling the operation every 3 minutes to create index...
[KFP Executor 2022-12-22 18:36:30,704 INFO]: Polling the operation every 3 minutes to create index...
[KFP Executor 2022-12-22 18:36:30,704 INFO]: Polling the operation every 3 minutes to create index...
[KFP Executor 2022-12-22 18:48:31,057 ERROR]: The index creation failed: MatchingEngineIndex resource has not been created. Resource failed with: Operation did not complete within the designated timeout of 900 seconds.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
api: vertex-aiIssues related to the googleapis/python-aiplatform API.Issues related to the googleapis/python-aiplatform API.
Activity
sasha-gitg commentedon Dec 29, 2022
@chrisk447 This seems to be the result of a breaking change in python-api-core that introduced a 900s default timeout on LROs: googleapis/python-api-core#462
Please try downgrading
google-api-core<2.11.0vam-google commentedon Jan 9, 2023
The old behavior was incorrect, there should be no assumption that core libraries would ever poll for an LRO indefinitely (old behavior). Please provide actual timeouts for any api-specific calls (i.e. something like
result(timeout = 60*60)for calls to PollingFuture. It is still possible to mimic the old infinite timeout behavior by calling it asresult(timeout = None)but please do not do it unless absolutely necessary, as infinite timeouts are an anti-pattern and allow writing code which may run for infinitely long.sasha-gitg commentedon Jan 13, 2023
Fixed in 1.21.
fix(components): Set google-cloud-aiplatform>=1.21.0 to ultimately av…
fix(components): Set google-cloud-aiplatform>=1.21.0 to ultimately av…
fix(components): Set google-cloud-aiplatform>=1.21.0 to ultimately av…
fix(components): Set google-cloud-aiplatform>=1.21.0 to ultimately av…
fix(components): Set google-cloud-aiplatform>=1.21.0 to ultimately av…
fix(components): Set google-cloud-aiplatform>=1.21.0 to ultimately av…
fix(components): Set google-cloud-aiplatform>=1.21.0 to ultimately av…
fix(components): Set google-cloud-aiplatform>=1.21.0 to ultimately av…
fix(components): Set google-cloud-aiplatform>=1.21.0 to ultimately av…