Skip to content

Commit 6cff4eb

Browse files
jlara310kurtisvg
jlara310
authored and
kurtisvg
committed
Add Appengine-Datastore scheduled export sample (GoogleCloudPlatform#1286)
* Initial commit of sample app for scheduling a datastore export with GAE. * Run java linter * Update README * Add a basic test for the schedule-datastore-export app * Update datastore/schedule-export/src/main/java/com/example/datastore/CloudDatastoreExport.java Co-Authored-By: jlara310 <1543140+jlara310@users.noreply.github.com> * Update datastore/schedule-export/src/main/java/com/example/datastore/CloudDatastoreExport.java Co-Authored-By: jlara310 <1543140+jlara310@users.noreply.github.com> * Add Licenses. Remove un-used code. Address style comments. * Read project ID from from environment variable. * Look for not OK 200 response first, and then for success * Address style guide violations * Use markdown style link * Use code fone for parameter list * Move app to App Engine folder and remove unused dependencies. * Remove unnecessary plugins
1 parent b4c6488 commit 6cff4eb

File tree

9 files changed

+404
-0
lines changed

9 files changed

+404
-0
lines changed

‎appengine-java8/datastore-schedule-export/README.md‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
# Scheduling a Cloud Datastore export
2+
3+
This Google App Engine (GAE) app receives export requests at `/cloud-datastore-export` and
4+
sends an export request to the [Cloud Datastore Admin API](https://cloud.google.com/datastore/docs/reference/admin/rest/v1/projects/export).
5+
6+
## Before you begin
7+
8+
This app requires the following to complete export operations:
9+
10+
1. A Google Cloud project with billing enabled.
11+
1. A Cloud Storage bucket for your export files.
12+
1. The App Engine default service account must have permission
13+
to write to the Cloud Storage bucket and have the Cloud Datastore Import Export Admin IAM role.
14+
15+
For more information on completing these requirements, see the
16+
[Cloud Datastore documentation](https://cloud.google.com/datastore/docs/schedule-export#before_you_begin).
17+
18+
## Deploying
19+
20+
Set the target project in gcloud:
21+
22+
gcloud config set project PROJECT_NAME
23+
24+
Deploy the GAE app:
25+
26+
mvn appengine:deploy
27+
28+
The app takes the following parameters:
29+
30+
* `output_url_prefix` (required)-specifies where to save your Cloud Datastore export. If the URL ends with a `/`, it's used as is. Otherwise, the app adds a timesamp to the url.
31+
* `kind` (optional, multiple)-restricts export to only these kinds.
32+
* `namespace_id` (optional, multiple)-restricts export to only these namespaces.
33+
34+
Modify and deploy the cronjob:
35+
36+
gcloud app deploy cron.yaml
37+
38+
You can test your cron job by running the job manually:
39+
40+
<a href="https://console.cloud.google.com/appengine/cronjobs">Open the Cron Jobs page</a>

‎appengine-java8/datastore-schedule-export/cron.yaml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
cron:
2+
- description: "Daily Cloud Datastore Export"
3+
url: /cloud-datastore-export?output_url_prefix=gs://<EXPORT_BUCKET>&kind=<KIND_NAME>
4+
schedule: every 24 hours

‎appengine-java8/datastore-schedule-export/pom.xml‎

Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<project>
3+
<modelVersion>4.0.0</modelVersion>
4+
<packaging>war</packaging>
5+
<version>1.0-SNAPSHOT</version>
6+
7+
<groupId>com.example.datastore</groupId>
8+
<artifactId>schedule-export</artifactId>
9+
10+
<parent>
11+
<groupId>com.google.cloud.samples</groupId>
12+
<artifactId>shared-configuration</artifactId>
13+
<version>1.0.10</version>
14+
</parent>
15+
16+
<properties>
17+
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
18+
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
19+
<maven.compiler.source>1.8</maven.compiler.source>
20+
<maven.compiler.target>1.8</maven.compiler.target>
21+
<maven.compiler.showDeprecation>true</maven.compiler.showDeprecation>
22+
<archiveClasses>true</archiveClasses>
23+
</properties>
24+
25+
<prerequisites>
26+
<maven>3.5</maven>
27+
</prerequisites>
28+
29+
<dependencies>
30+
<!-- Compile/runtime dependencies -->
31+
<dependency>
32+
<groupId>com.google.appengine</groupId>
33+
<artifactId>appengine-api-1.0-sdk</artifactId>
34+
<version>1.9.59</version>
35+
</dependency>
36+
<dependency>
37+
<groupId>com.google.guava</groupId>
38+
<artifactId>guava</artifactId>
39+
<version>25.1-android</version>
40+
</dependency>
41+
<dependency>
42+
<groupId>javax.servlet</groupId>
43+
<artifactId>javax.servlet-api</artifactId>
44+
<version>3.1.0</version>
45+
<type>jar</type>
46+
<scope>provided</scope>
47+
</dependency>
48+
<dependency>
49+
<groupId>org.json</groupId>
50+
<artifactId>json</artifactId>
51+
<version>20180130</version>
52+
</dependency>
53+
54+
<!-- Test Dependencies -->
55+
<dependency>
56+
<groupId>com.google.appengine</groupId>
57+
<artifactId>appengine-testing</artifactId>
58+
<version>1.9.59</version>
59+
<scope>test</scope>
60+
</dependency>
61+
<dependency>
62+
<groupId>com.google.truth</groupId>
63+
<artifactId>truth</artifactId>
64+
<version>0.33</version>
65+
<scope>test</scope>
66+
</dependency>
67+
<dependency>
68+
<groupId>junit</groupId>
69+
<artifactId>junit</artifactId>
70+
<version>4.12</version>
71+
<scope>test</scope>
72+
</dependency>
73+
<dependency>
74+
<groupId>org.mockito</groupId>
75+
<artifactId>mockito-all</artifactId>
76+
<version>2.0.2-beta</version>
77+
<scope>test</scope>
78+
</dependency>
79+
</dependencies>
80+
81+
<build>
82+
<!-- for hot reload of the web application-->
83+
<outputDirectory>${project.build.directory}/${project.build.finalName}/WEB-INF/classes</outputDirectory>
84+
<plugins>
85+
<plugin>
86+
<groupId>com.google.cloud.tools</groupId>
87+
<artifactId>appengine-maven-plugin</artifactId>
88+
<version>1.3.2</version>
89+
<configuration>
90+
</configuration>
91+
</plugin>
92+
</plugins>
93+
</build>
94+
</project>

‎appengine-java8/datastore-schedule-export/src/main/java/com/example/datastore/DatastoreExportServlet.java‎

Lines changed: 145 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,145 @@
1+
/*
2+
* Copyright 2018 Google LLC
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
package com.google.example.datastore;
18+
19+
import com.google.appengine.api.appidentity.AppIdentityService;
20+
import com.google.appengine.api.appidentity.AppIdentityServiceFactory;
21+
import com.google.apphosting.api.ApiProxy;
22+
import com.google.common.io.CharStreams;
23+
import java.io.IOException;
24+
import java.io.InputStream;
25+
import java.io.InputStreamReader;
26+
import java.io.OutputStreamWriter;
27+
import java.net.HttpURLConnection;
28+
import java.net.URL;
29+
import java.nio.charset.StandardCharsets;
30+
import java.text.SimpleDateFormat;
31+
import java.util.ArrayList;
32+
import java.util.Date;
33+
import java.util.logging.Logger;
34+
import javax.servlet.annotation.WebServlet;
35+
import javax.servlet.http.HttpServlet;
36+
import javax.servlet.http.HttpServletRequest;
37+
import javax.servlet.http.HttpServletResponse;
38+
import org.json.JSONArray;
39+
import org.json.JSONObject;
40+
import org.json.JSONTokener;
41+
42+
@WebServlet(name = "DatastoreExportServlet", value = "/cloud-datastore-export")
43+
public class DatastoreExportServlet extends HttpServlet {
44+
45+
private static final String PROJECT_ID = System.getenv("GOOGLE_CLOUD_PROJECT");
46+
47+
private static final Logger log = Logger.getLogger(DatastoreExportServlet.class.getName());
48+
49+
@Override
50+
public void doGet(HttpServletRequest request, HttpServletResponse response) throws IOException {
51+
52+
// Validate outputURL parameter
53+
String outputUrlPrefix = request.getParameter("output_url_prefix");
54+
55+
if (outputUrlPrefix == null || !outputUrlPrefix.matches("^gs://.*")) {
56+
// Send error response if outputURL not set or not a Cloud Storage bucket
57+
response.setStatus(HttpServletResponse.SC_CONFLICT);
58+
response.setContentType("text/plain");
59+
response.getWriter().println("Error: Must provide a valid output_url_prefix.");
60+
61+
} else {
62+
63+
// Put together export request headers
64+
URL url = new URL("https://datastore.googleapis.com/v1/projects/" + PROJECT_ID + ":export");
65+
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
66+
connection.setDoOutput(true);
67+
connection.setRequestMethod("POST");
68+
connection.addRequestProperty("Content-Type", "application/json");
69+
70+
// Get an access token to authorize export request
71+
ArrayList<String> scopes = new ArrayList<String>();
72+
scopes.add("https://www.googleapis.com/auth/datastore");
73+
final AppIdentityService appIdentity = AppIdentityServiceFactory.getAppIdentityService();
74+
final AppIdentityService.GetAccessTokenResult accessToken =
75+
AppIdentityServiceFactory.getAppIdentityService().getAccessToken(scopes);
76+
connection.addRequestProperty("Authorization", "Bearer " + accessToken.getAccessToken());
77+
78+
// Build export request payload based on URL parameters
79+
// Required: output_url_prefix
80+
// Optional: entity filter
81+
JSONObject exportRequest = new JSONObject();
82+
83+
// If output prefix ends with a slash, use as-is
84+
// Otherwise, add a timestamp to form unique output url
85+
if (!outputUrlPrefix.endsWith("/")) {
86+
String timeStamp = new SimpleDateFormat("yyyyMMddHHmmss").format(new Date());
87+
outputUrlPrefix = outputUrlPrefix + "/" + timeStamp + "/";
88+
}
89+
90+
// Add outputUrl to payload
91+
exportRequest.put("output_url_prefix", outputUrlPrefix);
92+
93+
// Build optional entity filter to export subset of
94+
// kinds or namespaces
95+
JSONObject entityFilter = new JSONObject();
96+
97+
// Read kind parameters and add to export request if not null
98+
String[] kinds = request.getParameterValues("kind");
99+
if (kinds != null) {
100+
JSONArray kindsJson = new JSONArray(kinds);
101+
entityFilter.put("kinds", kinds);
102+
}
103+
104+
// Read namespace parameters and add to export request if not null
105+
String[] namespaces = request.getParameterValues("namespace_id");
106+
if (namespaces != null) {
107+
JSONArray namespacesJson = new JSONArray(namespaces);
108+
entityFilter.put("namespaceIds", namespacesJson);
109+
}
110+
111+
// Add entity filter to payload
112+
// Finish export request payload
113+
exportRequest.put("entityFilter", entityFilter);
114+
115+
// Send export request
116+
OutputStreamWriter writer = new OutputStreamWriter(connection.getOutputStream());
117+
exportRequest.write(writer);
118+
writer.close();
119+
120+
// Examine server's response
121+
if (connection.getResponseCode() != HttpURLConnection.HTTP_OK) {
122+
// Request failed, log errors and return
123+
InputStream s = connection.getErrorStream();
124+
InputStreamReader r = new InputStreamReader(s, StandardCharsets.UTF_8);
125+
String errorMessage =
126+
String.format(
127+
"got error (%d) response %s from %s",
128+
connection.getResponseCode(), CharStreams.toString(r), connection.toString());
129+
log.warning(errorMessage);
130+
response.setStatus(HttpServletResponse.SC_INTERNAL_SERVER_ERROR);
131+
response.setContentType("text/plain");
132+
response.getWriter().println(
133+
"Failed to initiate export.");
134+
return;
135+
}
136+
137+
// Success, print export operation information
138+
JSONObject exportResponse = new JSONObject(new JSONTokener(connection.getInputStream()));
139+
140+
response.setContentType("text/plain");
141+
response.getWriter().println(
142+
"Export started:\n" + exportResponse.toString(4));
143+
}
144+
}
145+
}

‎appengine-java8/datastore-schedule-export/src/main/webapp/WEB-INF/appengine-web.xml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
<?xml version="1.0" encoding="utf-8"?>
2+
<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
3+
<runtime>java8</runtime>
4+
<threadsafe>true</threadsafe>
5+
<system-properties>
6+
<property name="java.util.logging.config.file" value="WEB-INF/logging.properties"/>
7+
</system-properties>
8+
</appengine-web-app>

‎appengine-java8/datastore-schedule-export/src/main/webapp/WEB-INF/logging.properties‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
# A default java.util.logging configuration.
2+
# (All App Engine logging is through java.util.logging by default).
3+
#
4+
# To use this configuration, copy it into your application's WEB-INF
5+
# folder and add the following to your appengine-web.xml:
6+
#
7+
# <system-properties>
8+
# <property name="java.util.logging.config.file" value="WEB-INF/logging.properties"/>
9+
# </system-properties>
10+
#
11+
12+
# Set the default logging level for all loggers to WARNING
13+
.level = WARNING

‎appengine-java8/datastore-schedule-export/src/main/webapp/WEB-INF/web.xml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
<?xml version="1.0" encoding="utf-8"?>
2+
<web-app xmlns="http://xmlns.jcp.org/xml/ns/javaee"
3+
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
4+
xsi:schemaLocation="http://xmlns.jcp.org/xml/ns/javaee
5+
http://xmlns.jcp.org/xml/ns/javaee/web-app_3_1.xsd" version="3.1">
6+
<welcome-file-list>
7+
<welcome-file>index.jsp</welcome-file>
8+
</welcome-file-list>
9+
</web-app>

‎appengine-java8/datastore-schedule-export/src/main/webapp/index.jsp‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
<!DOCTYPE html>
2+
<%@ page contentType="text/html;charset=UTF-8" language="java" %>
3+
<html>
4+
<head>
5+
<link href='//fonts.googleapis.com/css?family=Marmelad' rel='stylesheet' type='text/css'>
6+
<title>Hello App Engine Standard Java 8</title>
7+
</head>
8+
<body>
9+
<h1>Hello App Engine!</h1>
10+
</body>
11+
</html>

0 commit comments

Comments
 (0)