Skip to content

Commit bdda289

Browse files
rogerthatdevpattishingcf-owl-bot[bot]kweinmeisteraverikitsch
authored
feat(run): add Cloud Run + Filestore sample (GoogleCloudPlatform#3288)
* initial package.json * basic express with test * add testing dependencies * add nonexistant path test * refactor test to use chai expect as assert * add chai to package.json * hide console.log output from test output * label console output * semi-colons ;;;; * add redirects to mount dir * writeFile function * test for new file write * return list of files and links * add test for files * add Dockerfile and wrapper script * add header * eslint --fix * add rate limit to app * add testing workflows * dont store value for path * add missing #! * refactor to use serve-index * eslint * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Update run/filesystem/package.json Co-authored-by: Averi Kitsch <akitsch@google.com> * Update run/filesystem/package.json Co-authored-by: Averi Kitsch <akitsch@google.com> * remove fs from package.json * typo * move port var down * change filename format * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * refactor: redirect all to mnt path * eslint * additional comments * separate function for index * eslint * fix hrefs * move file prefix var * period * rename wrapper script * update wrapper * comments and sample description * cross-site scripting * add try catch * better error response * sanitize res * use node:20-slim * rm unused dep * arrow functions and consts * eslint * no mutations * unused var * remove unneeded async * lint * add sigterm handling * fix sigterm handling * formatting * e2e test cloudbuild config * parallel delete resources * fix cleanup step * add waitfor * break cleanup into 3 steps * clean up AR Repo build * split build config into setup and cleanup * e2e test boilerplate * remove unused subs from cleanup yaml * update waitFor in cleanup yaml * recombine build yamls * lint * split cloudbuild config * split tests to system.test.js * lint * system.test add auth * add before and after to e2e test * add endpoint test to e2e * update package.json * headers * typo * add to system e2e tests * kokoro config * remove cfg * typo * remove server.close * refactor system.test.js to match other samples * add test for generated txt to e2e * add comment re: rate limit * remove allow unauthenticated * lint: unused var * add unit test for file generation * clarify rate limit comment * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * add wait -n to run script * use fs/promises * write file before generating html --------- Co-authored-by: Patti Shin <pattishin@users.noreply.github.com> Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: Karl Weinmeister <11586922+kweinmeister@users.noreply.github.com> Co-authored-by: Averi Kitsch <akitsch@google.com>
1 parent 735eea7 commit bdda289

File tree

10 files changed

+553
-0
lines changed

10 files changed

+553
-0
lines changed
Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
# Copyright 2023 Google LLC
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
name: run-filesystem
16+
on:
17+
push:
18+
branches:
19+
- main
20+
paths:
21+
- 'run/filesystem/**'
22+
- '.github/workflows/run-filesystem.yaml'
23+
pull_request:
24+
paths:
25+
- 'run/filesystem/**'
26+
- '.github/workflows/run-filesystem.yaml'
27+
pull_request_target:
28+
types: [labeled]
29+
paths:
30+
- 'run/filesystem/**'
31+
- '.github/workflows/run-filesystem.yaml'
32+
schedule:
33+
- cron: '0 0 * * 0'
34+
jobs:
35+
test:
36+
# Ref: https://github.com/google-github-actions/auth#usage
37+
permissions:
38+
contents: 'read'
39+
id-token: 'write'
40+
if: github.event.action != 'labeled' || github.event.label.name == 'actions:force-run'
41+
uses: ./.github/workflows/test.yaml
42+
with:
43+
name: 'run-filesystem'
44+
path: 'run/filesystem'
45+
remove_label:
46+
# Ref: https://github.com/google-github-actions/auth#usage
47+
permissions:
48+
contents: 'read'
49+
id-token: 'write'
50+
if: |
51+
github.event.action == 'labeled' &&
52+
github.event.label.name == 'actions:force-run' &&
53+
always()
54+
uses: ./.github/workflows/remove-label.yaml
55+
flakybot:
56+
# Ref: https://github.com/google-github-actions/auth#usage
57+
permissions:
58+
contents: 'read'
59+
id-token: 'write'
60+
if: github.event_name == 'schedule' && always() # always() submits logs even if tests fail
61+
uses: ./.github/workflows/flakybot.yaml
62+
needs: [test]

.github/workflows/utils/workflows.json

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -85,6 +85,7 @@
8585
"monitoring/snippets",
8686
"opencensus",
8787
"retail",
88+
"run/filesystem",
8889
"scheduler",
8990
"secret-manager",
9091
"service-directory/snippets",

run/filesystem/Dockerfile

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
# Copyright 2023 Google LLC
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
# [START cloudrun_fs_dockerfile]
16+
17+
# Use the official Node.js image.
18+
# https://hub.docker.com/_/node
19+
FROM node:20-slim
20+
21+
# Install system dependencies
22+
RUN apt-get update -y && apt-get install -y \
23+
tini \
24+
nfs-common \
25+
libtool \
26+
&& apt-get clean
27+
28+
# Set fallback mount directory
29+
ENV MNT_DIR /mnt/nfs/filestore
30+
31+
# Copy local code to the container image.
32+
ENV APP_HOME /app
33+
WORKDIR $APP_HOME
34+
COPY package*.json ./
35+
36+
# Install production dependencies.
37+
RUN npm install --only=production
38+
39+
# Copy local code to the container image.
40+
COPY . ./
41+
42+
# Ensure the script is executable
43+
RUN chmod +x /app/run.sh
44+
45+
# Use tini to manage zombie processes and signal forwarding
46+
# https://github.com/krallin/tini
47+
ENTRYPOINT ["/usr/bin/tini", "--"]
48+
49+
# Pass the wrapper script as arguments to tini
50+
CMD ["/app/run.sh"]
51+
# [END cloudrun_fs_dockerfile]

run/filesystem/index.js

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
// Copyright 2023 Google LLC
2+
//
3+
// Licensed under the Apache License, Version 2.0 (the "License");
4+
// you may not use this file except in compliance with the License.
5+
// You may obtain a copy of the License at
6+
//
7+
// https://www.apache.org/licenses/LICENSE-2.0
8+
//
9+
// Unless required by applicable law or agreed to in writing, software
10+
// distributed under the License is distributed on an "AS IS" BASIS,
11+
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
// See the License for the specific language governing permissions and
13+
// limitations under the License.
14+
/**
15+
* Express webapp that generates files to a mounted NFS when deployed to Cloud Run.
16+
*
17+
* See https://cloud.google.com/run/docs/tutorials/network-filesystems-filestore before running the code snippet.
18+
*/
19+
20+
const express = require('express');
21+
const rateLimit = require('express-rate-limit');
22+
const fs = require('fs/promises');
23+
const app = express();
24+
const mntDir = process.env.MNT_DIR || '/mnt/nfs/filestore';
25+
const port = parseInt(process.env.PORT) || 8080;
26+
const limit = rateLimit({
27+
// Use of rate limit to fullfill CodeQL rule js/missing-rate-limiting
28+
// HTTP request handlers should not perform expensive operations such as
29+
// accessing the file system. Setting rate limit to maximum 100 requests
30+
// per 15 minute window.
31+
windowMs: 15 * 60 * 1000,
32+
max: 100,
33+
message: 'Rate limit exceeded',
34+
headers: true,
35+
});
36+
37+
app.use(limit);
38+
app.use(mntDir, express.static(mntDir));
39+
40+
app.listen(port, () => {
41+
console.log(`Listening on port ${port}`);
42+
});
43+
44+
app.get(mntDir, async (req, res) => {
45+
// Have all requests to mount directory generate a new file on the filesystem.
46+
try {
47+
writeFile(mntDir);
48+
const html = await generateIndex(mntDir);
49+
// Respond with html with list of files on the filesystem.
50+
res.send(html);
51+
} catch (error) {
52+
console.error(error);
53+
res
54+
.status(500)
55+
.send(
56+
'Something went wrong when writing to the filesystem. Refresh the page to try again.'
57+
);
58+
}
59+
});
60+
61+
app.all('*', (req, res) => {
62+
// Redirect all requests to the mount directory
63+
res.redirect(mntDir);
64+
});
65+
66+
const writeFile = async (path, filePrefix = 'test') => {
67+
// Write a test file to the provided path.
68+
const date = new Date();
69+
const formattedDate = date.toString().split(' ').slice(0, 5).join('-');
70+
const filename = `${filePrefix}-${formattedDate}.txt`;
71+
const contents = `This test file was created on ${formattedDate}.\n`;
72+
73+
try {
74+
const newFile = fs.writeFile(`${path}/${filename}`, contents);
75+
await newFile;
76+
} catch (error) {
77+
console.error(error);
78+
}
79+
};
80+
81+
const generateIndex = async mntDir => {
82+
// Return html for page with a list of files on the mounted filesystem.
83+
try {
84+
const header =
85+
'<html><body>A new file is generated each time this page is reloaded.<p>Files created on filesystem:<p>';
86+
const footer = '</body></html>';
87+
// Get list of files on mounted filesystem.
88+
const existingFiles = await fs.readdir(mntDir);
89+
// Insert each file into html content
90+
const htmlBody = existingFiles.map(fileName => {
91+
const sanitized = encodeURIComponent(fileName);
92+
return `<a href="${mntDir}/${sanitized}">${decodeURIComponent(
93+
sanitized
94+
)}</a><br>`;
95+
});
96+
return header + htmlBody.join(' ') + footer;
97+
} catch (error) {
98+
console.log(error);
99+
}
100+
};
101+
102+
process.on('SIGTERM', () => {
103+
console.log('Received SIGTERM signal. Exiting.');
104+
});
105+
106+
module.exports = app;

run/filesystem/package.json

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
{
2+
"name": "filesystem",
3+
"description": "Demonstrate accessing a mounted network filesystem from a Cloud Run service.",
4+
"version": "1.0.0",
5+
"main": "index.js",
6+
"scripts": {
7+
"start": "node index.js",
8+
"test": "c8 mocha test/index.test.js --exit",
9+
"system-test": "c8 mocha test/system.test.js --timeout=360000 --exit"
10+
},
11+
"engines": {
12+
"node": ">=12.0.0"
13+
},
14+
"author": "Google LLC",
15+
"license": "Apache-2.0",
16+
"dependencies": {
17+
"express": "^4.18.2",
18+
"express-rate-limit": "^6.7.0"
19+
},
20+
"devDependencies": {
21+
"c8": "^7.14.0",
22+
"chai": "^4.3.7",
23+
"chai-http": "^4.4.0",
24+
"google-auth-library": "^8.0.0",
25+
"got": "^11.0.0",
26+
"mocha": "^10.2.0",
27+
"mock-fs": "^5.2.0",
28+
"supertest": "^6.3.3"
29+
}
30+
}

run/filesystem/run.sh

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
#!/usr/bin/env bash
2+
# Copyright 2023 Google LLC
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License");
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
16+
# [START cloudrun_fs_script]
17+
#!/usr/bin/env bash
18+
set -eo pipefail
19+
20+
# Create mount directory for service.
21+
mkdir -p $MNT_DIR
22+
23+
echo "Mounting Cloud Filestore."
24+
mount -o nolock $FILESTORE_IP_ADDRESS:/$FILE_SHARE_NAME $MNT_DIR
25+
echo "Mounting completed."
26+
27+
# Start the application
28+
node index.js
29+
30+
# Exit immediately when one of the background processes terminate.
31+
wait -n
32+
# [END cloudrun_fs_script]
Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
# Copyright 2023 Google LLC
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# https://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
steps:
16+
- id: 'Delete Cloud Run service'
17+
name: 'gcr.io/cloud-builders/gcloud:latest'
18+
entrypoint: /bin/bash
19+
waitFor: ['-']
20+
args:
21+
- '-c'
22+
- |
23+
gcloud run services delete $_RUN_SERVICE --region $_REGION --quiet
24+
25+
- id: 'Delete Artifact Registry build'
26+
name: 'gcr.io/cloud-builders/gcloud:latest'
27+
entrypoint: /bin/bash
28+
waitFor: ['-']
29+
args:
30+
- '-c'
31+
- |
32+
gcloud artifacts docker images delete \
33+
us-central1-docker.pkg.dev/$PROJECT_ID/cloud-run-source-deploy/$_RUN_SERVICE:latest --quiet
34+
35+
substitutions:
36+
_REGION: us-central1
37+
_RUN_SERVICE: filesystem-app
38+
Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
# Copyright 2023 Google LLC
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# https://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
steps:
16+
- id: 'Build and deploy to Cloud Run'
17+
name: 'gcr.io/cloud-builders/gcloud:latest'
18+
entrypoint: /bin/bash
19+
args:
20+
- '-c'
21+
- |
22+
gcloud run deploy $_RUN_SERVICE --source . \
23+
--region $_REGION \
24+
--vpc-connector $_CONNECTOR_NAME \
25+
--execution-environment gen2 \
26+
--update-env-vars FILESTORE_IP_ADDRESS=$_FILESTORE_IP_ADDRESS,FILE_SHARE_NAME=$_SHARE_NAME
27+
28+
substitutions:
29+
_REGION: us-central1
30+
_FILESTORE_IP_ADDRESS: 10.103.89.66 # Existing long-standing resource
31+
_SHARE_NAME: filestoresamples # Existing long-standing resource
32+
_CONNECTOR_NAME: run-filesystem-e2e-test # Existing long-standing resource
33+
_RUN_SERVICE: filesystem-app
34+

0 commit comments

Comments
 (0)