Skip to content

Commit e9ecfac

Browse files
author
Doug Greiman
committed
Update build process.
build.sh is the entrypoint for all builds. cloudbuild*.yaml files are now the single source of truth for the build steps. Makefiles have been removed. Local builds are done via local_cloudbuild.py. jenkins_build.sh will be removed once the Jenkins config is updated. The benchmarking and system tests now work in Container Builder. Build instructions have been moved into their own file. gcloud substitutions are used for cloudbuild.yaml substitions. See 'gcloud beta container builds submit --help' for more info. 'envsubst' is still used for Dockerifle substitutions. Updated the google-cloud-python test logic.
1 parent c960a87 commit e9ecfac

36 files changed

Lines changed: 400 additions & 342 deletions

‎.gitignore‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
1-
/cloudbuild.yaml
1+
*.pyc
2+
.nox
23
/cloudbuild.yaml_local.sh
3-
/ext_run.sh
4+
/cloudbuild_benchmark.yaml_local.sh
5+
/cloudbuild_system_tests.yaml_local.sh
46
__pycache__
5-
.nox
6-
*.pyc

‎Makefile‎

Lines changed: 0 additions & 73 deletions
This file was deleted.

‎README.md‎

Lines changed: 1 addition & 67 deletions
Original file line numberDiff line numberDiff line change
@@ -72,73 +72,7 @@ command or entrypoint. For example:
7272
Google regularly builds and releases this image at
7373
[`gcr.io/google-appengine/python`](https://gcr.io/google-appengine/python).
7474

75-
To rebuild the image yourself, first set the following variables in your
76-
shell. You need to be authenticated to a Google Cloud Project to invoke the
77-
Google Container Builder service, and also to run the system tests.
78-
79-
```shell
80-
$ export GOOGLE_CLOUD_PROJECT=YOUR-PROJECT-NAME
81-
$ DOCKER_NAMESPACE=gcr.io/${GCLOUD_PROJECT}
82-
$ CANDIDATE_NAME=`date +%Y-%m-%d_%H_%M`
83-
$ export IMAGE_NAME=${DOCKER_NAMESPACE}/python:${CANDIDATE_NAME}
84-
$ gcloud config set project ${GOOGLE_CLOUD_PROJECT}
85-
```
86-
87-
To rebuild the image using the Google Container Builder service, do the
88-
following:
89-
90-
```shell
91-
$ make cloud-build
92-
$ make cloud-test
93-
```
94-
95-
To rebuild the image using your local Docker daemon, do the following:
96-
97-
``` shell
98-
$ make local-build
99-
$ make local-test
100-
```
101-
102-
To open an interactive shell session to this image after building it, do the following:
103-
104-
``` shell
105-
docker run -it --entrypoint /bin/bash ${IMAGE_NAME}
106-
```
107-
108-
## Running the system tests
109-
110-
To run the system tests, you need a Google Cloud Project with a service account.
111-
From the [Google Cloud Console](https://console.cloud.google.com/), either
112-
create a new project or switch to an existing one. Next,
113-
[create a service account](
114-
https://cloud.google.com/iam/docs/creating-managing-service-accounts) that will
115-
be used to run the system tests. Once you have a service account,
116-
[create and download a service account key](https://cloud.google.com/iam/docs/managing-service-account-keys).
117-
118-
In the
119-
[IAM & Admin](https://console.cloud.google.com/permissions/projectpermissions)
120-
section, grant the `Owner` role to the service account you created above. Also
121-
grant the `Editor` role to the `cloud-logs@google.com` service account.
122-
123-
124-
Then, follow the
125-
[system test setup instructions](https://github.com/GoogleCloudPlatform/google-cloud-python/blob/master/CONTRIBUTING.rst#running-system-tests). It
126-
describes various steps, including running some scripts to populate and/or
127-
delete datastore example data and indexes (populate_datastore.py,
128-
clear_datastore.py, and `gcloud preview datastore create-indexes
129-
system_tests/data/index.yaml`).
130-
131-
From the cloud console, you will need to enable the following APIs for your project:
132-
133-
- Bigquery API
134-
- Cloud Bigtable Admin API
135-
- Google Cloud Natural Language API
136-
- Google Cloud Pub/Sub API
137-
- Google Cloud Storage JSON API
138-
- Google Cloud Vision API
139-
- Google Translate API
140-
- Stackdriver Logging API
141-
- Stackdriver Monitoring API
75+
See [RELEASING.md](RELEASING.md) for more information.
14276

14377
## Contributing changes
14478

‎RELEASING.md‎

Lines changed: 130 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,130 @@
1+
# Google Cloud Platform - Python Runtime Docker Image
2+
3+
## `build.sh`
4+
5+
There is a shell script called `build.sh` that builds everything in this
6+
repository.
7+
8+
### Environment variables for `build.sh`
9+
10+
DOCKER_NAMESPACE
11+
: The prefix applied to all images names created. To push images to Google
12+
Container Registry (GCR), this should be `gcr.io/YOUR-PROJECT-NAME`.
13+
14+
TAG
15+
: The suffix applied to all images created. This should be unique. If not
16+
specified, the current time will be used (timestamp format `YYYY-mm-dd-HHMMSS`).
17+
18+
GOOGLE_APPLICATION_CREDENTIALS
19+
: (System test only) Path to service account credentials in JSON format.
20+
21+
GOOGLE_CLOUD_PROJECT
22+
: (System test only) Name of the Google Cloud Platform project to run the system
23+
tests under.
24+
25+
## Building and Releasing
26+
27+
A custom Jenkins job builds and releases this repository using scripts and job
28+
configurations that are not yet available publicly. The control flow is as
29+
follows:
30+
31+
1. Jenkins job `python/release` is invoked by
32+
a. Manually running the script `build_and_release.py` with arguments
33+
b. Manually invoking the job from the GUI
34+
2. The job runs the script `release.sh`
35+
a. Service account credentials are read
36+
b. `gcloud auth activate-service-account` is performed
37+
c. `gcloud config set project` is performed
38+
3. The script invokes `build.sh` in this repository
39+
4. `build.sh` invokes Google Container Builder with the `cloudbuild-*.yaml`
40+
config files.
41+
42+
## Building outside Jenkins
43+
44+
To build this repository outside Jenkins, authenticate and authorize yourself
45+
with `gcloud auth`, set the variables listed above, and run:
46+
47+
``` shell
48+
./build.sh
49+
```
50+
51+
This assumes an environment similar to the internal Jenkins environment (Linux,
52+
Debian or Ubuntu-like).
53+
54+
## Building locally
55+
56+
To build this repository using local Docker commands instead of the Google
57+
Container Builder service, add the `--local` flag as shown:
58+
59+
``` shell
60+
./build.sh --local
61+
```
62+
63+
To open an interactive shell session to this image after building it, do the
64+
following:
65+
66+
``` shell
67+
docker run -it --entrypoint /bin/bash YOUR-IMAGE-NAME
68+
```
69+
70+
## Running benchmarks
71+
72+
There is a benchmark suite which compares the performance of interpreters
73+
against each other.
74+
75+
``` shell
76+
./build.sh --nobuild --benchmark
77+
```
78+
79+
Since these benchmarks are run on cloud instances, the timings may vary from run
80+
to run.
81+
82+
## Running system tests
83+
84+
**TAKE NOTE: You will incur charges for use of Google Cloud Platform services!**
85+
86+
System tests perform mutating operations against the real Google Cloud services.
87+
Since these system tests may fail or be flaky for outside reasons such as
88+
netorking issues, configuration errors, or services outages, they are run
89+
separately from building the images, and should be run in their own project.
90+
91+
To run the system tests, you need a Google Cloud Project with a service account.
92+
From the [Google Cloud Console](https://console.cloud.google.com/), either
93+
create a new project or switch to an existing one. Next,
94+
[create a service account](
95+
https://cloud.google.com/iam/docs/creating-managing-service-accounts) that will
96+
be used to run the system tests. Once you have a service account,
97+
[create and download a service account key](https://cloud.google.com/iam/docs/managing-service-account-keys).
98+
99+
In the
100+
[IAM & Admin](https://console.cloud.google.com/permissions/projectpermissions)
101+
section, grant the `Owner` role to the service account you created above. Also
102+
grant the `Editor` role to the `cloud-logs@google.com` service account.
103+
104+
Then, follow the
105+
[system test setup instructions](https://github.com/GoogleCloudPlatform/google-cloud-python/blob/master/CONTRIBUTING.rst#running-system-tests). It
106+
describes various steps, including running some scripts to populate and/or
107+
delete datastore example data and indexes (populate_datastore.py,
108+
clear_datastore.py, and `gcloud preview datastore create-indexes
109+
system_tests/data/index.yaml`).
110+
111+
From the cloud console, you will need to enable at least the following APIs for
112+
your project:
113+
114+
- Bigquery API
115+
- Cloud Bigtable Admin API
116+
- Cloud Spanner API
117+
- Google Cloud Natural Language API
118+
- Google Cloud Pub/Sub API
119+
- Google Cloud Speech API
120+
- Google Cloud Storage JSON API
121+
- Google Cloud Translation API
122+
- Google Cloud Vision API
123+
- Stackdriver Logging API
124+
- Stackdriver Monitoring API
125+
126+
Once all the setup has been done, run the following:
127+
128+
``` shell
129+
./build.sh --nobuild --system_tests
130+
```

0 commit comments

Comments
 (0)