Skip to content

Commit 9347913

Browse files
kweinmeisterkweinmeister
authored
fix: add Retail retries (GoogleCloudPlatform#9722)
1 parent 48169c5 commit 9347913

26 files changed

Lines changed: 176 additions & 69 deletions

‎retail/interactive-tutorials/README.md‎

Lines changed: 77 additions & 43 deletions
Original file line numberDiff line numberDiff line change
@@ -2,51 +2,64 @@
22

33
## Run tutorials in Cloud Shell
44

5-
To advance with the interactive tutorials, use Retail Search step-by-step manuals on the right side of the Cloud Shell IDE:
6-
![Interactive tutorials](images/tutorial1.png)
5+
To advance with the interactive tutorials, use Retail Search step-by-step
6+
manuals on the right side of the Cloud Shell IDE: ![Interactive
7+
tutorials](images/tutorial1.png)
78

8-
The interactive tutorial should open by default. If it didn’t, click on the Tutorial symbol in the menu bar to open the step-by-step manual:
9-
![Toggle tutorial](images/tutorials2.png)
9+
The interactive tutorial should open by default. If it didn’t, click on the
10+
Tutorial symbol in the menu bar to open the step-by-step manual: ![Toggle
11+
tutorial](images/tutorials2.png)
1012

11-
For more details about the Cloud Shell environment, refer to the [Cloud Shell documentation](https://cloud.google.com/shell/docs).
13+
For more details about the Cloud Shell environment, refer to the [Cloud Shell
14+
documentation](https://cloud.google.com/shell/docs).
1215

1316
## Interactive tutorial flow
1417

15-
Interactive guides are intended to help you understand the features provided by Google Cloud Retail Search and test the Retail API in action.
18+
Interactive guides are intended to help you understand the features provided by
19+
Google Cloud Retail Search and test the Retail API in action.
1620

17-
To proceed with the tutorial, choose a language you’ll be deploying your project in:
18-
![Select a programming language](images/tutorials3.png)
21+
To proceed with the tutorial, choose a language you’ll be deploying your project
22+
in: ![Select a programming language](images/tutorials3.png)
1923

20-
21-
To begin with the tutorial workflow, click the Start button:
22-
![Begin with the tutorial](images/tutorials4.png)
24+
To begin with the tutorial workflow, click the Start button: ![Begin with the
25+
tutorial](images/tutorials4.png)
2326

2427
Then, you can use Next and Previous buttons to navigate the tutorial pages.
2528

2629
## Get started with the Google Cloud Retail API
2730

28-
The Retail API provides you with the following possibilities to:
29-
- Create and maintaining the catalog data.
30-
- Fine-tune the search configuration.
31-
- Import and maintain the user events data.
31+
The Retail API provides you with the following possibilities to:
32+
33+
- Create and maintaining the catalog data.
34+
- Fine-tune the search configuration.
35+
- Import and maintain the user events data.
3236

33-
You can find the information about the Retail services in the [documentation](https://cloud.google.com/retail/docs)
37+
You can find the information about the Retail services in the
38+
[documentation](https://cloud.google.com/retail/docs)
3439

3540
<!--TODO(tkarasova) update the link to the tutorials when will be published-->
36-
If you would like to have a closer look at the Retail API features and try them yourself,
37-
the best option is to use the [Interactive Tutorials](https://cloud.google.com/retail/docs/overview). The tutorials will be launched in the CloudShell environment, and you will be able to request the Retail services and check the response with minimum time and effort.
41+
If you would like to have a closer look at the Retail API features and try them
42+
yourself, the best option is to use the [Interactive
43+
Tutorials](https://cloud.google.com/retail/docs/overview). The tutorials will be
44+
launched in the CloudShell environment, and you will be able to request the
45+
Retail services and check the response with minimum time and effort.
3846

39-
The code samples in the directory **python-docs-samples/retail/interactive-tutorials** are explicitly created for use with the Retail Interactive Tutorials.
47+
The code samples in the directory
48+
**python-docs-samples/retail/interactive-tutorials** are explicitly created for
49+
use with the Retail Interactive Tutorials.
4050

41-
If, for some reason, you have decided to proceed with these code samples without the tutorial, please go through the following steps and set up the required preconditions.
51+
If, for some reason, you have decided to proceed with these code samples without
52+
the tutorial, please go through the following steps and set up the required
53+
preconditions.
4254

4355
### Select your project and enable the Retail API
4456

45-
Google Cloud organizes resources into projects. This lets you
46-
collect all related resources for a single application in one place.
57+
Google Cloud organizes resources into projects. This lets you collect all
58+
related resources for a single application in one place.
4759

48-
If you don't have a Google Cloud project yet or you're not the owner of an existing one, you can
49-
[create a new project](https://console.cloud.google.com/projectcreate).
60+
If you don't have a Google Cloud project yet or you're not the owner of an
61+
existing one, you can [create a new
62+
project](https://console.cloud.google.com/projectcreate).
5063

5164
After the project is created, set your PROJECT_ID to a ```project``` variable:
5265

@@ -56,18 +69,23 @@ After the project is created, set your PROJECT_ID to a ```project``` variable:
5669
gcloud config set project <YOUR_PROJECT_ID>
5770
```
5871

59-
1. Ensure that the Retail API is enabled for your project in the [API & Services page](https://console.cloud.google.com/apis/api/retail.googleapis.com/).
72+
1. Ensure that the Retail API is enabled for your project in the [API & Services
73+
page](https://console.cloud.google.com/apis/api/retail.googleapis.com/).
6074

6175
## Prepare your work environment
6276

6377
To prepare the work environment you should perform the following steps:
78+
6479
- Create a service account.
65-
- Create service account key and set it to authorize your calls to the Retail API.
80+
- Create service account key and set it to authorize your calls to the Retail
81+
API.
6682
- Install Google Cloud Retail library.
6783

6884
### Create service account
6985

70-
To access the Retail API, you must create a service account. Check that you are an owner of your Google Cloud project on the [IAM page](https://console.cloud.google.com/iam-admin/iam).
86+
To access the Retail API, you must create a service account. Check that you are
87+
an owner of your Google Cloud project on the [IAM
88+
page](https://console.cloud.google.com/iam-admin/iam).
7189

7290
1. To create a service account, perform the following command:
7391

@@ -91,8 +109,7 @@ To access the Retail API, you must create a service account. Check that you are
91109

92110
Copy the service account email.
93111

94-
95-
1. Upload your service account key JSON file and use it to activate the service
112+
1. Upload your service account key JSON file and use it to activate the service
96113
account:
97114

98115
```bash
@@ -103,18 +120,20 @@ To access the Retail API, you must create a service account. Check that you are
103120
gcloud auth activate-service-account --key-file ~/key.json
104121
```
105122

106-
1. Set the key as the GOOGLE_APPLICATION_CREDENTIALS environment variable to
107-
use it for sending requests to the Retail API.
123+
1. Set the key as the GOOGLE_APPLICATION_CREDENTIALS environment variable to use
124+
it for sending requests to the Retail API.
108125

109126
```bash
110127
export GOOGLE_APPLICATION_CREDENTIALS=~/key.json
111128
```
112129

113130
### Install Google Cloud Retail libraries
114131

115-
To run Python code samples for the Retail API tutorial, you need to set up your virtual environment.
132+
To run Python code samples for the Retail API tutorial, you need to set up your
133+
virtual environment.
116134

117-
1. Run the following commands in a Terminal to create an isolated Python environment:
135+
1. Run the following commands in a Terminal to create an isolated Python
136+
environment:
118137

119138
```bash
120139
virtualenv -p python3 myenv
@@ -133,36 +152,48 @@ To run Python code samples for the Retail API tutorial, you need to set up your
133152

134153
## Import catalog data
135154

136-
There is a python-docs-samples/retail/interactive-tutorials/resources/products.json file with valid products prepared in the `resources` directory.
155+
There is a
156+
python-docs-samples/retail/interactive-tutorials/resources/products.json file
157+
with valid products prepared in the `resources` directory.
137158

138-
The other file, python-docs-samples/retail/interactive-tutorials/resources/products_some_invalid.json, contains both valid and invalid products. You will use it to check the error handling.
159+
The other file,
160+
python-docs-samples/retail/interactive-tutorials/resources/products_some_invalid.json,
161+
contains both valid and invalid products. You will use it to check the error
162+
handling.
139163

140164
### Upload catalog data to Cloud Storage
141165

142-
In your own project you need to create a Cloud Storage bucket and put the JSON file there.
143-
The bucket name must be unique. For convenience, you can name it `<YOUR_PROJECT_ID>_<TIMESTAMP>`.
166+
In your own project you need to create a Cloud Storage bucket and put the JSON
167+
file there. The bucket name must be unique. For convenience, you can name it
168+
`<YOUR_PROJECT_ID>_<TIMESTAMP>`.
144169

145-
1. The code samples for each of the Retail services are stored in different directories.
170+
1. The code samples for each of the Retail services are stored in different
171+
directories.
146172

147173
Go to the code samples directory, your starting point to run more commands.
148174

149175
```bash
150176
cd python-docs-samples/retail/interactive-tutorials
151177
```
152178

153-
1. To create the bucket and upload the JSON file, open python-docs-samples/retail/interactive-tutorials/product/setup_product/products_create_gcs_bucket.py file
179+
1. To create the bucket and upload the JSON file, open
180+
python-docs-samples/retail/interactive-tutorials/product/setup_product/products_create_gcs_bucket.py
181+
file
154182

155-
1. Go to the **product** directory and run the following command in the Terminal:
183+
1. Go to the **product** directory and run the following command in the
184+
Terminal:
156185

157186
```bash
158187
python product/setup_product/products_create_gcs_bucket.py
159188
```
160189

161-
Now you can see the bucket is created in the [Cloud Storage](https://console.cloud.google.com/storage/browser), and the files are uploaded.
190+
Now you can see the bucket is created in the [Cloud
191+
Storage](https://console.cloud.google.com/storage/browser), and the files
192+
are uploaded.
162193

163194
1. The name of the created Cloud Storage bucket is printed in the Terminal.
164195

165-
```
196+
```bash
166197
The gcs bucket <YOUR_PROJECT_ID>_<TIMESTAMP> was created
167198
```
168199

@@ -174,7 +205,9 @@ The bucket name must be unique. For convenience, you can name it `<YOUR_PROJECT_
174205

175206
### Import products to the Retail Catalog
176207

177-
To import the prepared products to a catalog, open python-docs-samples/retail/interactive-tutorials/product/import_products_gcs.py file and run the following command in the Terminal:
208+
To import the prepared products to a catalog, open
209+
python-docs-samples/retail/interactive-tutorials/product/import_products_gcs.py
210+
file and run the following command in the Terminal:
178211

179212
```bash
180213
python product/import_products_gcs.py
@@ -183,6 +216,7 @@ python product/import_products_gcs.py
183216
## Run your code sample
184217

185218
Run the sample in a terminal with the following command:
219+
186220
```bash
187221
python search/search_with_boost_spec.py
188222
```

‎retail/interactive-tutorials/TEST_RESOURCES_SETUP_CLEANUP.md‎

Lines changed: 27 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -3,11 +3,12 @@
33
## Required environment variables
44

55
To successfully import the catalog data for tests, the following environment variables should be set:
6-
- BUCKET_NAME
7-
- EVENTS_BUCKET_NAME
8-
These values are stored in the Secret Manager and will be submitted as
6+
7+
- BUCKET_NAME
8+
- EVENTS_BUCKET_NAME
9+
These values are stored in the Secret Manager and will be submitted as
910
docker environment variables before the test run.
10-
11+
1112
The Secret Manager name is set in .kokoro/presubmit/common.cfg file, SECRET_MANAGER_KEYS variable.
1213

1314
## Import catalog data
@@ -16,35 +17,35 @@ There are JSON files with valid products and user events prepared in `resources`
1617
`samples/resources/products.json` and `samples/resources/user_events.json` respectively.
1718

1819
Run the `create_test_resources.py` to perform the following actions:
19-
- create the GCS bucket <BUCKET_NAME>,
20-
- upload the product data from `resources/products.json` file to products bucket,
21-
- import products to the default branch of the Retail catalog,
22-
- create the GCS bucket <EVENTS_BUCKET_NAME>,
23-
- upload the product data from `resources/user_events.json` file to events bucket,
24-
- create a BigQuery dataset and table `products`,
25-
- insert products from resources/products.json to the created products table,
26-
- create a BigQuery dataset and table `events`,
27-
- insert user events from resources/user_events.json to the created events table
28-
2920

30-
```
31-
$ python create_test_resources.py
21+
- create the GCS bucket <BUCKET_NAME>,
22+
- upload the product data from `resources/products.json` file to products bucket,
23+
- import products to the default branch of the Retail catalog,
24+
- create the GCS bucket <EVENTS_BUCKET_NAME>,
25+
- upload the product data from `resources/user_events.json` file to events bucket,
26+
- create a BigQuery dataset and table `products`,
27+
- insert products from resources/products.json to the created products table,
28+
- create a BigQuery dataset and table `events`,
29+
- insert user events from resources/user_events.json to the created events table
30+
31+
```sh
32+
python create_test_resources.py
3233
```
3334

3435
In the result 316 products should be created in the test project catalog.
3536

36-
3737
## Remove catalog data
3838

3939
Run the `remove_test_resources.py` to perform the following actions:
40-
- remove all objects from the GCS bucket <BUCKET_NAME>,
41-
- remove the <BUCKET_NAME> bucket,
42-
- delete all products from the Retail catalog.
43-
- remove all objects from the GCS bucket <EVENTS_BUCKET_NAME>,
44-
- remove the <EVENTS_BUCKET_NAME> bucket,
45-
- remove dataset `products` along with tables
46-
- remove dataset `user_events` along with tables
4740

41+
- remove all objects from the GCS bucket <BUCKET_NAME>,
42+
- remove the <BUCKET_NAME> bucket,
43+
- delete all products from the Retail catalog.
44+
- remove all objects from the GCS bucket <EVENTS_BUCKET_NAME>,
45+
- remove the <EVENTS_BUCKET_NAME> bucket,
46+
- remove dataset `products` along with tables
47+
- remove dataset `user_events` along with tables
48+
49+
```sh
50+
python remove_test_resources.py
4851
```
49-
$ python remove_test_resources.py
50-
```

‎retail/interactive-tutorials/events/import_user_events_bigquery_test.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,8 @@
1515
import re
1616
import subprocess
1717

18+
from google.api_core.retry import Retry
19+
1820
from setup_events.setup_cleanup import (
1921
create_bq_dataset,
2022
create_bq_table,
@@ -24,6 +26,7 @@
2426
from setup_events.update_user_events_json import update_events_timestamp
2527

2628

29+
@Retry()
2730
def test_import_products_bq(table_id_prefix):
2831
dataset = "user_events"
2932
valid_products_table = f"{table_id_prefix}events"

‎retail/interactive-tutorials/events/import_user_events_gcs_test.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,10 +15,13 @@
1515
import re
1616
import subprocess
1717

18+
from google.api_core.retry import Retry
19+
1820
from setup_events.setup_cleanup import create_bucket, delete_bucket, upload_blob
1921
from setup_events.update_user_events_json import update_events_timestamp
2022

2123

24+
@Retry()
2225
def test_import_events_gcs(bucket_name_prefix):
2326
# gcs buckets have a limit of 63 characters. Get the last 60 characters
2427
bucket_name = bucket_name_prefix[63:]

‎retail/interactive-tutorials/events/import_user_events_inline_test.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,10 @@
1515
import re
1616
import subprocess
1717

18+
from google.api_core.retry import Retry
1819

20+
21+
@Retry()
1922
def test_create_product():
2023
output = str(
2124
subprocess.check_output("python import_user_events_inline.py", shell=True)

‎retail/interactive-tutorials/events/purge_user_event_test.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,10 @@
1515
import re
1616
import subprocess
1717

18+
from google.api_core.retry import Retry
1819

20+
21+
@Retry()
1922
def test_create_product():
2023
output = str(subprocess.check_output("python purge_user_event.py", shell=True))
2124

‎retail/interactive-tutorials/events/rejoin_user_event_test.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,10 @@
1515
import re
1616
import subprocess
1717

18+
from google.api_core.retry import Retry
1819

20+
21+
@Retry()
1922
def test_create_product():
2023
output = str(subprocess.check_output("python rejoin_user_event.py", shell=True))
2124

‎retail/interactive-tutorials/events/write_user_event_test.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,10 @@
1515
import re
1616
import subprocess
1717

18+
from google.api_core.retry import Retry
1819

20+
21+
@Retry()
1922
def test_create_product():
2023
output = str(subprocess.check_output("python write_user_event.py", shell=True))
2124

‎retail/interactive-tutorials/product/add_fulfillment_places_test.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,10 @@
1515
import re
1616
import subprocess
1717

18+
from google.api_core.retry import Retry
1819

20+
21+
@Retry()
1922
def test_add_fulfillment():
2023
output = str(
2124
subprocess.check_output("python add_fulfillment_places.py", shell=True)

0 commit comments

Comments
 (0)