From 9b48326a19109644404445c7c50ea0ed6860c4d6 Mon Sep 17 00:00:00 2001 From: "release-please[bot]" <55107282+release-please[bot]@users.noreply.github.com> Date: Mon, 3 Oct 2022 20:50:17 +0000 Subject: [PATCH 01/11] chore(main): release 2.11.6-SNAPSHOT (#868) :robot: I have created a release *beep* *boop* --- ### Updating meta-information for bleeding-edge SNAPSHOT release. --- This PR was generated with [Release Please](https://togithub.com/googleapis/release-please). See [documentation](https://togithub.com/googleapis/release-please#release-please). --- datastore-v1-proto-client/pom.xml | 4 ++-- google-cloud-datastore-bom/pom.xml | 10 +++++----- google-cloud-datastore/pom.xml | 4 ++-- grpc-google-cloud-datastore-admin-v1/pom.xml | 4 ++-- pom.xml | 12 ++++++------ proto-google-cloud-datastore-admin-v1/pom.xml | 4 ++-- proto-google-cloud-datastore-v1/pom.xml | 4 ++-- versions.txt | 12 ++++++------ 8 files changed, 27 insertions(+), 27 deletions(-) diff --git a/datastore-v1-proto-client/pom.xml b/datastore-v1-proto-client/pom.xml index d8662ff57..028c2d459 100644 --- a/datastore-v1-proto-client/pom.xml +++ b/datastore-v1-proto-client/pom.xml @@ -19,12 +19,12 @@ 4.0.0 com.google.cloud.datastore datastore-v1-proto-client - 2.11.5 + 2.11.6-SNAPSHOT com.google.cloud google-cloud-datastore-parent - 2.11.5 + 2.11.6-SNAPSHOT jar diff --git a/google-cloud-datastore-bom/pom.xml b/google-cloud-datastore-bom/pom.xml index 9c5186304..74d2a04e1 100644 --- a/google-cloud-datastore-bom/pom.xml +++ b/google-cloud-datastore-bom/pom.xml @@ -3,7 +3,7 @@ 4.0.0 com.google.cloud google-cloud-datastore-bom - 2.11.5 + 2.11.6-SNAPSHOT pom com.google.cloud @@ -52,22 +52,22 @@ com.google.cloud google-cloud-datastore - 2.11.5 + 2.11.6-SNAPSHOT com.google.api.grpc grpc-google-cloud-datastore-admin-v1 - 2.11.5 + 2.11.6-SNAPSHOT com.google.api.grpc proto-google-cloud-datastore-v1 - 0.102.5 + 0.102.6-SNAPSHOT com.google.api.grpc proto-google-cloud-datastore-admin-v1 - 2.11.5 + 2.11.6-SNAPSHOT diff --git a/google-cloud-datastore/pom.xml b/google-cloud-datastore/pom.xml index d5a22486e..81698ae66 100644 --- a/google-cloud-datastore/pom.xml +++ b/google-cloud-datastore/pom.xml @@ -2,7 +2,7 @@ 4.0.0 google-cloud-datastore - 2.11.5 + 2.11.6-SNAPSHOT jar Google Cloud Datastore https://github.com/googleapis/java-datastore @@ -12,7 +12,7 @@ com.google.cloud google-cloud-datastore-parent - 2.11.5 + 2.11.6-SNAPSHOT google-cloud-datastore diff --git a/grpc-google-cloud-datastore-admin-v1/pom.xml b/grpc-google-cloud-datastore-admin-v1/pom.xml index 4da271bb8..b6805ba68 100644 --- a/grpc-google-cloud-datastore-admin-v1/pom.xml +++ b/grpc-google-cloud-datastore-admin-v1/pom.xml @@ -4,13 +4,13 @@ 4.0.0 com.google.api.grpc grpc-google-cloud-datastore-admin-v1 - 2.11.5 + 2.11.6-SNAPSHOT grpc-google-cloud-datastore-admin-v1 GRPC library for google-cloud-datastore com.google.cloud google-cloud-datastore-parent - 2.11.5 + 2.11.6-SNAPSHOT diff --git a/pom.xml b/pom.xml index ef3df4089..2c53584b3 100644 --- a/pom.xml +++ b/pom.xml @@ -4,7 +4,7 @@ com.google.cloud google-cloud-datastore-parent pom - 2.11.5 + 2.11.6-SNAPSHOT Google Cloud Datastore Parent https://github.com/googleapis/java-datastore @@ -159,27 +159,27 @@ com.google.api.grpc proto-google-cloud-datastore-admin-v1 - 2.11.5 + 2.11.6-SNAPSHOT com.google.api.grpc grpc-google-cloud-datastore-admin-v1 - 2.11.5 + 2.11.6-SNAPSHOT com.google.cloud google-cloud-datastore - 2.11.5 + 2.11.6-SNAPSHOT com.google.api.grpc proto-google-cloud-datastore-v1 - 0.102.5 + 0.102.6-SNAPSHOT com.google.cloud.datastore datastore-v1-proto-client - 2.11.5 + 2.11.6-SNAPSHOT com.google.api.grpc diff --git a/proto-google-cloud-datastore-admin-v1/pom.xml b/proto-google-cloud-datastore-admin-v1/pom.xml index 2bbfb1874..684d38fdf 100644 --- a/proto-google-cloud-datastore-admin-v1/pom.xml +++ b/proto-google-cloud-datastore-admin-v1/pom.xml @@ -4,13 +4,13 @@ 4.0.0 com.google.api.grpc proto-google-cloud-datastore-admin-v1 - 2.11.5 + 2.11.6-SNAPSHOT proto-google-cloud-datastore-admin-v1 Proto library for google-cloud-datastore com.google.cloud google-cloud-datastore-parent - 2.11.5 + 2.11.6-SNAPSHOT diff --git a/proto-google-cloud-datastore-v1/pom.xml b/proto-google-cloud-datastore-v1/pom.xml index 17edc539e..47a2cc7b9 100644 --- a/proto-google-cloud-datastore-v1/pom.xml +++ b/proto-google-cloud-datastore-v1/pom.xml @@ -4,13 +4,13 @@ 4.0.0 com.google.api.grpc proto-google-cloud-datastore-v1 - 0.102.5 + 0.102.6-SNAPSHOT proto-google-cloud-datastore-v1 PROTO library for proto-google-cloud-datastore-v1 com.google.cloud google-cloud-datastore-parent - 2.11.5 + 2.11.6-SNAPSHOT diff --git a/versions.txt b/versions.txt index fac256bee..be559a86e 100644 --- a/versions.txt +++ b/versions.txt @@ -1,9 +1,9 @@ # Format: # module:released-version:current-version -google-cloud-datastore:2.11.5:2.11.5 -google-cloud-datastore-bom:2.11.5:2.11.5 -proto-google-cloud-datastore-v1:0.102.5:0.102.5 -datastore-v1-proto-client:2.11.5:2.11.5 -proto-google-cloud-datastore-admin-v1:2.11.5:2.11.5 -grpc-google-cloud-datastore-admin-v1:2.11.5:2.11.5 +google-cloud-datastore:2.11.5:2.11.6-SNAPSHOT +google-cloud-datastore-bom:2.11.5:2.11.6-SNAPSHOT +proto-google-cloud-datastore-v1:0.102.5:0.102.6-SNAPSHOT +datastore-v1-proto-client:2.11.5:2.11.6-SNAPSHOT +proto-google-cloud-datastore-admin-v1:2.11.5:2.11.6-SNAPSHOT +grpc-google-cloud-datastore-admin-v1:2.11.5:2.11.6-SNAPSHOT From 15eddd98d4468445b9e989a027f8a842b50d5ec3 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Tue, 4 Oct 2022 18:54:15 +0200 Subject: [PATCH 02/11] chore(deps): update dependency com.google.cloud:google-cloud-datastore to v2.11.5 (#869) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit [![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [com.google.cloud:google-cloud-datastore](https://togithub.com/googleapis/java-datastore) | `2.11.4` -> `2.11.5` | [![age](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-datastore/2.11.5/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-datastore/2.11.5/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-datastore/2.11.5/compatibility-slim/2.11.4)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-datastore/2.11.5/confidence-slim/2.11.4)](https://docs.renovatebot.com/merge-confidence/) | --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Renovate will not automatically rebase this PR, because other commits have been found. 🔕 **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] If you want to rebase/retry this PR, click this checkbox. ⚠ **Warning**: custom changes will be lost. --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/java-datastore). --- README.md | 6 +++--- samples/install-without-bom/pom.xml | 2 +- samples/snapshot/pom.xml | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 65642fcfb..95360cb59 100644 --- a/README.md +++ b/README.md @@ -41,7 +41,7 @@ If you are using Maven without BOM, add this to your dependencies: com.google.cloud google-cloud-datastore - 2.11.4 + 2.11.5 ``` @@ -56,13 +56,13 @@ implementation 'com.google.cloud:google-cloud-datastore' If you are using Gradle without BOM, add this to your dependencies: ```Groovy -implementation 'com.google.cloud:google-cloud-datastore:2.11.4' +implementation 'com.google.cloud:google-cloud-datastore:2.11.5' ``` If you are using SBT, add this to your dependencies: ```Scala -libraryDependencies += "com.google.cloud" % "google-cloud-datastore" % "2.11.4" +libraryDependencies += "com.google.cloud" % "google-cloud-datastore" % "2.11.5" ``` ## Authentication diff --git a/samples/install-without-bom/pom.xml b/samples/install-without-bom/pom.xml index 0cfeca427..1fbcaff77 100644 --- a/samples/install-without-bom/pom.xml +++ b/samples/install-without-bom/pom.xml @@ -29,7 +29,7 @@ com.google.cloud google-cloud-datastore - 2.11.4 + 2.11.5 diff --git a/samples/snapshot/pom.xml b/samples/snapshot/pom.xml index 05064b415..fb1e20988 100644 --- a/samples/snapshot/pom.xml +++ b/samples/snapshot/pom.xml @@ -28,7 +28,7 @@ com.google.cloud google-cloud-datastore - 2.11.4 + 2.11.5 From 4a8da722d5a5c1614b11f80a84fbf1754f47d5df Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Fri, 7 Oct 2022 17:46:29 +0200 Subject: [PATCH 03/11] chore(deps): update dependency com.google.cloud:libraries-bom to v26.1.3 (#870) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * chore(deps): update dependency com.google.cloud:libraries-bom to v26.1.3 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md Co-authored-by: Owl Bot --- README.md | 4 ++-- samples/native-image-sample/pom.xml | 2 +- samples/snippets/pom.xml | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 95360cb59..fca5470e0 100644 --- a/README.md +++ b/README.md @@ -19,7 +19,7 @@ If you are using Maven with [BOM][libraries-bom], add this to your pom.xml file: com.google.cloud libraries-bom - 26.1.2 + 26.1.3 pom import @@ -49,7 +49,7 @@ If you are using Maven without BOM, add this to your dependencies: If you are using Gradle 5.x or later, add this to your dependencies: ```Groovy -implementation platform('com.google.cloud:libraries-bom:26.1.2') +implementation platform('com.google.cloud:libraries-bom:26.1.3') implementation 'com.google.cloud:google-cloud-datastore' ``` diff --git a/samples/native-image-sample/pom.xml b/samples/native-image-sample/pom.xml index 07a9283e4..aab2b9174 100644 --- a/samples/native-image-sample/pom.xml +++ b/samples/native-image-sample/pom.xml @@ -28,7 +28,7 @@ com.google.cloud libraries-bom - 26.1.2 + 26.1.3 pom import diff --git a/samples/snippets/pom.xml b/samples/snippets/pom.xml index b1e99e22a..995c8800b 100644 --- a/samples/snippets/pom.xml +++ b/samples/snippets/pom.xml @@ -30,7 +30,7 @@ com.google.cloud libraries-bom - 26.1.2 + 26.1.3 pom import From b2a72ca407b1fa168c18b136e73932c8716fbdf6 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Tue, 11 Oct 2022 21:57:34 +0200 Subject: [PATCH 04/11] deps: update dependency com.google.errorprone:error_prone_core to v2.16 (#872) --- pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pom.xml b/pom.xml index 2c53584b3..3ed672f27 100644 --- a/pom.xml +++ b/pom.xml @@ -143,7 +143,7 @@ github google-cloud-datastore-parent https://googleapis.dev/java/google-api-grpc/latest - 2.15.0 + 2.16 From 0e4ab63f04b97df62d59420d0009ded11610ad2d Mon Sep 17 00:00:00 2001 From: Prateek Date: Fri, 14 Oct 2022 18:22:19 +0530 Subject: [PATCH 05/11] refactor: making query class generic (#875) * Making query generic and moving all the entity records related concerns to an internal RecordQuery interface * fixing lint * Making getType an internal function --- .../google/cloud/datastore/DatastoreImpl.java | 4 ++- .../com/google/cloud/datastore/GqlQuery.java | 22 ++++++++++--- .../com/google/cloud/datastore/Query.java | 16 ++------- .../cloud/datastore/QueryResultsImpl.java | 11 ++++--- .../google/cloud/datastore/RecordQuery.java | 33 +++++++++++++++++++ .../cloud/datastore/StructuredQuery.java | 19 ++++++++--- .../google/cloud/datastore/DatastoreTest.java | 4 +-- 7 files changed, 79 insertions(+), 30 deletions(-) create mode 100644 google-cloud-datastore/src/main/java/com/google/cloud/datastore/RecordQuery.java diff --git a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/DatastoreImpl.java b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/DatastoreImpl.java index 9892e1517..2d68a2294 100644 --- a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/DatastoreImpl.java +++ b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/DatastoreImpl.java @@ -180,8 +180,10 @@ public QueryResults run(Query query, ReadOption... options) { return run(toReadOptionsPb(options), query); } + @SuppressWarnings("unchecked") QueryResults run(com.google.datastore.v1.ReadOptions readOptionsPb, Query query) { - return new QueryResultsImpl<>(this, readOptionsPb, query); + return new QueryResultsImpl<>( + this, readOptionsPb, (RecordQuery) query, query.getNamespace()); } com.google.datastore.v1.RunQueryResponse runQuery( diff --git a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/GqlQuery.java b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/GqlQuery.java index 2b99fd0a9..bebb4df9f 100644 --- a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/GqlQuery.java +++ b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/GqlQuery.java @@ -19,6 +19,7 @@ import static com.google.cloud.datastore.Validator.validateNamespace; import static com.google.common.base.Preconditions.checkNotNull; +import com.google.api.core.InternalApi; import com.google.cloud.Timestamp; import com.google.common.base.MoreObjects; import com.google.common.collect.ImmutableList; @@ -71,7 +72,7 @@ * @param the type of the result values this query will produce * @see GQL Reference */ -public final class GqlQuery extends Query { +public final class GqlQuery extends Query implements RecordQuery { private static final long serialVersionUID = -5514894742849230793L; @@ -80,6 +81,8 @@ public final class GqlQuery extends Query { private final ImmutableMap namedBindings; private final ImmutableList positionalBindings; + private final ResultType resultType; + static final class Binding implements Serializable { private static final long serialVersionUID = 2344746877591371548L; @@ -423,7 +426,8 @@ private static Binding toBinding( } private GqlQuery(Builder builder) { - super(builder.resultType, builder.namespace); + super(builder.namespace); + resultType = checkNotNull(builder.resultType); queryString = builder.queryString; allowLiteral = builder.allowLiteral; namedBindings = ImmutableMap.copyOf(builder.namedBindings); @@ -461,9 +465,15 @@ public List getNumberArgs() { return builder.build(); } + @Override + public ResultType getType() { + return resultType; + } + @Override public String toString() { - return super.toStringHelper() + return toStringHelper() + .add("type", getType()) .add("queryString", queryString) .add("allowLiteral", allowLiteral) .add("namedBindings", namedBindings) @@ -507,13 +517,15 @@ com.google.datastore.v1.GqlQuery toPb() { return queryPb.build(); } + @InternalApi @Override - void populatePb(com.google.datastore.v1.RunQueryRequest.Builder requestPb) { + public void populatePb(com.google.datastore.v1.RunQueryRequest.Builder requestPb) { requestPb.setGqlQuery(toPb()); } + @InternalApi @Override - Query nextQuery(com.google.datastore.v1.RunQueryResponse responsePb) { + public RecordQuery nextQuery(com.google.datastore.v1.RunQueryResponse responsePb) { return StructuredQuery.fromPb(getType(), getNamespace(), responsePb.getQuery()) .nextQuery(responsePb); } diff --git a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/Query.java b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/Query.java index 00aa6f17c..a0bed5984 100644 --- a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/Query.java +++ b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/Query.java @@ -16,8 +16,6 @@ package com.google.cloud.datastore; -import static com.google.common.base.Preconditions.checkNotNull; - import com.google.common.base.MoreObjects; import com.google.common.base.MoreObjects.ToStringHelper; import com.google.common.collect.Maps; @@ -39,7 +37,6 @@ public abstract class Query implements Serializable { private static final long serialVersionUID = 7967659059395653941L; - private final ResultType resultType; private final String namespace; /** @@ -156,27 +153,18 @@ static ResultType fromPb(com.google.datastore.v1.EntityResult.ResultType type } } - Query(ResultType resultType, String namespace) { - this.resultType = checkNotNull(resultType); + Query(String namespace) { this.namespace = namespace; } - ResultType getType() { - return resultType; - } - public String getNamespace() { return namespace; } ToStringHelper toStringHelper() { - return MoreObjects.toStringHelper(this).add("type", resultType).add("namespace", namespace); + return MoreObjects.toStringHelper(this).add("namespace", namespace); } - abstract void populatePb(com.google.datastore.v1.RunQueryRequest.Builder requestPb); - - abstract Query nextQuery(com.google.datastore.v1.RunQueryResponse responsePb); - /** * Returns a new {@link GqlQuery} builder. * diff --git a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/QueryResultsImpl.java b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/QueryResultsImpl.java index 9ed822985..222efa3b8 100644 --- a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/QueryResultsImpl.java +++ b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/QueryResultsImpl.java @@ -30,7 +30,7 @@ class QueryResultsImpl extends AbstractIterator implements QueryResults private final com.google.datastore.v1.ReadOptions readOptionsPb; private final com.google.datastore.v1.PartitionId partitionIdPb; private final ResultType queryResultType; - private Query query; + private RecordQuery query; private ResultType actualResultType; private com.google.datastore.v1.RunQueryResponse runQueryResponsePb; private com.google.datastore.v1.Query mostRecentQueryPb; @@ -40,7 +40,10 @@ class QueryResultsImpl extends AbstractIterator implements QueryResults private MoreResultsType moreResults; QueryResultsImpl( - DatastoreImpl datastore, com.google.datastore.v1.ReadOptions readOptionsPb, Query query) { + DatastoreImpl datastore, + com.google.datastore.v1.ReadOptions readOptionsPb, + RecordQuery query, + String namespace) { this.datastore = datastore; this.readOptionsPb = readOptionsPb; this.query = query; @@ -48,8 +51,8 @@ class QueryResultsImpl extends AbstractIterator implements QueryResults com.google.datastore.v1.PartitionId.Builder pbBuilder = com.google.datastore.v1.PartitionId.newBuilder(); pbBuilder.setProjectId(datastore.getOptions().getProjectId()); - if (query.getNamespace() != null) { - pbBuilder.setNamespaceId(query.getNamespace()); + if (namespace != null) { + pbBuilder.setNamespaceId(namespace); } else if (datastore.getOptions().getNamespace() != null) { pbBuilder.setNamespaceId(datastore.getOptions().getNamespace()); } diff --git a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/RecordQuery.java b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/RecordQuery.java new file mode 100644 index 000000000..9dc966457 --- /dev/null +++ b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/RecordQuery.java @@ -0,0 +1,33 @@ +/* + * Copyright 2022 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.google.cloud.datastore; + +import com.google.api.core.InternalApi; +import com.google.cloud.datastore.Query.ResultType; + +/** An internal marker interface to represent {@link Query} that returns the entity records. */ +@InternalApi +public interface RecordQuery { + + @InternalApi + ResultType getType(); + + @InternalApi + void populatePb(com.google.datastore.v1.RunQueryRequest.Builder requestPb); + + @InternalApi + RecordQuery nextQuery(com.google.datastore.v1.RunQueryResponse responsePb); +} diff --git a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/StructuredQuery.java b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/StructuredQuery.java index 8e50d0867..8d2974c15 100644 --- a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/StructuredQuery.java +++ b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/StructuredQuery.java @@ -26,6 +26,7 @@ import static com.google.common.base.Preconditions.checkNotNull; import com.google.api.core.ApiFunction; +import com.google.api.core.InternalApi; import com.google.cloud.StringEnumType; import com.google.cloud.StringEnumValue; import com.google.cloud.Timestamp; @@ -85,7 +86,7 @@ * @see Datastore * queries */ -public abstract class StructuredQuery extends Query { +public abstract class StructuredQuery extends Query implements RecordQuery { private static final long serialVersionUID = 546838955624019594L; static final String KEY_PROPERTY_NAME = "__key__"; @@ -100,6 +101,8 @@ public abstract class StructuredQuery extends Query { private final int offset; private final Integer limit; + private final ResultType resultType; + public abstract static class Filter implements Serializable { private static final long serialVersionUID = -6443285436239990860L; @@ -899,7 +902,8 @@ B mergeFrom(com.google.datastore.v1.Query queryPb) { } StructuredQuery(BuilderImpl builder) { - super(builder.resultType, builder.namespace); + super(builder.namespace); + resultType = checkNotNull(builder.resultType); kind = builder.kind; projection = ImmutableList.copyOf(builder.projection); filter = builder.filter; @@ -914,6 +918,7 @@ B mergeFrom(com.google.datastore.v1.Query queryPb) { @Override public String toString() { return toStringHelper() + .add("type", getType()) .add("kind", kind) .add("startCursor", startCursor) .add("endCursor", endCursor) @@ -1013,13 +1018,19 @@ public Integer getLimit() { public abstract Builder toBuilder(); + public ResultType getType() { + return resultType; + } + + @InternalApi @Override - void populatePb(com.google.datastore.v1.RunQueryRequest.Builder requestPb) { + public void populatePb(com.google.datastore.v1.RunQueryRequest.Builder requestPb) { requestPb.setQuery(toPb()); } + @InternalApi @Override - StructuredQuery nextQuery(com.google.datastore.v1.RunQueryResponse responsePb) { + public StructuredQuery nextQuery(com.google.datastore.v1.RunQueryResponse responsePb) { Builder builder = toBuilder(); builder.setStartCursor(new Cursor(responsePb.getBatch().getEndCursor())); if (offset > 0 && responsePb.getBatch().getSkippedResults() < offset) { diff --git a/google-cloud-datastore/src/test/java/com/google/cloud/datastore/DatastoreTest.java b/google-cloud-datastore/src/test/java/com/google/cloud/datastore/DatastoreTest.java index fa077bc61..72067fd20 100644 --- a/google-cloud-datastore/src/test/java/com/google/cloud/datastore/DatastoreTest.java +++ b/google-cloud-datastore/src/test/java/com/google/cloud/datastore/DatastoreTest.java @@ -613,7 +613,7 @@ private List buildResponsesForQueryPagination() { Entity entity5 = Entity.newBuilder(KEY5).set("value", "value").build(); datastore.add(ENTITY3, entity4, entity5); List responses = new ArrayList<>(); - Query query = Query.newKeyQueryBuilder().build(); + RecordQuery query = Query.newKeyQueryBuilder().build(); RunQueryRequest.Builder requestPb = RunQueryRequest.newBuilder(); query.populatePb(requestPb); QueryResultBatch queryResultBatchPb = @@ -722,7 +722,7 @@ private List buildResponsesForQueryPaginationWithLimit() { datastore.add(ENTITY3, entity4, entity5); DatastoreRpc datastoreRpc = datastore.getOptions().getDatastoreRpcV1(); List responses = new ArrayList<>(); - Query query = Query.newEntityQueryBuilder().build(); + RecordQuery query = Query.newEntityQueryBuilder().build(); RunQueryRequest.Builder requestPb = RunQueryRequest.newBuilder(); query.populatePb(requestPb); QueryResultBatch queryResultBatchPb = From 4641306b1df99b7b637bb146af03e6bcb5ebf59f Mon Sep 17 00:00:00 2001 From: Prateek Date: Fri, 14 Oct 2022 19:16:13 +0530 Subject: [PATCH 06/11] refactor: Extracting out Proto preparation logic to encourage reusability (#876) This change introduces a component called ProtoPreparer which will be used to populate the protos from the domain objects. In the upcoming count aggregation feature, aggregation query will be acting upon StructuredQuery and GqlQuery and hence need to populate the protos of those queries **again**. For that reason _ProtoPreparer_ component will have the responsibility to populate the protos and be reused in multiple usecases. _Note: This PR is intended to trim down the changes done as part of #823 ._ --- google-cloud-datastore/pom.xml | 6 + .../google/cloud/datastore/DatastoreImpl.java | 52 +++----- .../com/google/cloud/datastore/GqlQuery.java | 23 ++-- .../datastore/GqlQueryProtoPreparer.java | 43 +++++++ .../cloud/datastore/QueryResultsImpl.java | 10 +- .../google/cloud/datastore/ReadOption.java | 44 +++++++ .../datastore/ReadOptionProtoPreparer.java | 77 ++++++++++++ .../cloud/datastore/StructuredQuery.java | 36 +----- .../StructuredQueryProtoPreparer.java | 66 ++++++++++ .../cloud/datastore/TransactionImpl.java | 22 ++-- .../execution/request/ProtoPreparer.java | 30 +++++ .../datastore/GqlQueryProtoPreparerTest.java | 74 +++++++++++ .../google/cloud/datastore/ProtoTestData.java | 76 +++++++++++ .../ReadOptionProtoPreparerTest.java | 97 ++++++++++++++ .../StructuredQueryProtoPreparerTest.java | 118 ++++++++++++++++++ 15 files changed, 679 insertions(+), 95 deletions(-) create mode 100644 google-cloud-datastore/src/main/java/com/google/cloud/datastore/GqlQueryProtoPreparer.java create mode 100644 google-cloud-datastore/src/main/java/com/google/cloud/datastore/ReadOptionProtoPreparer.java create mode 100644 google-cloud-datastore/src/main/java/com/google/cloud/datastore/StructuredQueryProtoPreparer.java create mode 100644 google-cloud-datastore/src/main/java/com/google/cloud/datastore/execution/request/ProtoPreparer.java create mode 100644 google-cloud-datastore/src/test/java/com/google/cloud/datastore/GqlQueryProtoPreparerTest.java create mode 100644 google-cloud-datastore/src/test/java/com/google/cloud/datastore/ProtoTestData.java create mode 100644 google-cloud-datastore/src/test/java/com/google/cloud/datastore/ReadOptionProtoPreparerTest.java create mode 100644 google-cloud-datastore/src/test/java/com/google/cloud/datastore/StructuredQueryProtoPreparerTest.java diff --git a/google-cloud-datastore/pom.xml b/google-cloud-datastore/pom.xml index 81698ae66..27ca86ab5 100644 --- a/google-cloud-datastore/pom.xml +++ b/google-cloud-datastore/pom.xml @@ -143,6 +143,12 @@ easymock test + + com.google.truth + truth + 1.1.3 + test + diff --git a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/DatastoreImpl.java b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/DatastoreImpl.java index 2d68a2294..f6a96c2a4 100644 --- a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/DatastoreImpl.java +++ b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/DatastoreImpl.java @@ -22,8 +22,6 @@ import com.google.cloud.RetryHelper; import com.google.cloud.RetryHelper.RetryHelperException; import com.google.cloud.ServiceOptions; -import com.google.cloud.datastore.ReadOption.EventualConsistency; -import com.google.cloud.datastore.ReadOption.ReadTime; import com.google.cloud.datastore.spi.v1.DatastoreRpc; import com.google.common.base.MoreObjects; import com.google.common.base.Preconditions; @@ -31,7 +29,7 @@ import com.google.common.collect.ImmutableList; import com.google.common.collect.Iterables; import com.google.common.collect.Sets; -import com.google.datastore.v1.ReadOptions.ReadConsistency; +import com.google.datastore.v1.ReadOptions; import com.google.datastore.v1.ReserveIdsRequest; import com.google.datastore.v1.TransactionOptions; import com.google.protobuf.ByteString; @@ -46,6 +44,7 @@ import java.util.LinkedHashSet; import java.util.List; import java.util.Map; +import java.util.Optional; import java.util.Set; import java.util.concurrent.Callable; @@ -57,13 +56,16 @@ final class DatastoreImpl extends BaseService implements Datas TransactionExceptionHandler.build(); private static final ExceptionHandler TRANSACTION_OPERATION_EXCEPTION_HANDLER = TransactionOperationExceptionHandler.build(); - private final TraceUtil traceUtil = TraceUtil.getInstance();; + private final TraceUtil traceUtil = TraceUtil.getInstance(); + + private final ReadOptionProtoPreparer readOptionProtoPreparer; DatastoreImpl(DatastoreOptions options) { super(options); this.datastoreRpc = options.getDatastoreRpcV1(); retrySettings = MoreObjects.firstNonNull(options.getRetrySettings(), ServiceOptions.getNoRetrySettings()); + readOptionProtoPreparer = new ReadOptionProtoPreparer(); } @Override @@ -172,7 +174,7 @@ public T runInTransaction( @Override public QueryResults run(Query query) { - return run(null, query); + return run(Optional.empty(), query); } @Override @@ -181,7 +183,7 @@ public QueryResults run(Query query, ReadOption... options) { } @SuppressWarnings("unchecked") - QueryResults run(com.google.datastore.v1.ReadOptions readOptionsPb, Query query) { + QueryResults run(Optional readOptionsPb, Query query) { return new QueryResultsImpl<>( this, readOptionsPb, (RecordQuery) query, query.getNamespace()); } @@ -331,7 +333,7 @@ public Entity get(Key key, ReadOption... options) { @Override public Iterator get(Key... keys) { - return get(null, keys); + return get(Optional.empty(), keys); } @Override @@ -339,33 +341,11 @@ public Iterator get(Iterable keys, ReadOption... options) { return get(toReadOptionsPb(options), Iterables.toArray(keys, Key.class)); } - private static com.google.datastore.v1.ReadOptions toReadOptionsPb(ReadOption... options) { - com.google.datastore.v1.ReadOptions readOptionsPb = null; - if (options != null) { - Map, ReadOption> optionsByType = - ReadOption.asImmutableMap(options); - - if (optionsByType.containsKey(EventualConsistency.class) - && optionsByType.containsKey(ReadTime.class)) { - throw DatastoreException.throwInvalidRequest( - "Can not use eventual consistency read with read time."); - } - - if (optionsByType.containsKey(EventualConsistency.class)) { - readOptionsPb = - com.google.datastore.v1.ReadOptions.newBuilder() - .setReadConsistency(ReadConsistency.EVENTUAL) - .build(); - } - - if (optionsByType.containsKey(ReadTime.class)) { - readOptionsPb = - com.google.datastore.v1.ReadOptions.newBuilder() - .setReadTime(((ReadTime) optionsByType.get(ReadTime.class)).time().toProto()) - .build(); - } + private Optional toReadOptionsPb(ReadOption... options) { + if (options == null) { + return Optional.empty(); } - return readOptionsPb; + return this.readOptionProtoPreparer.prepare(Arrays.asList(options)); } @Override @@ -378,15 +358,13 @@ public List fetch(Iterable keys, ReadOption... options) { return DatastoreHelper.fetch(this, Iterables.toArray(keys, Key.class), options); } - Iterator get(com.google.datastore.v1.ReadOptions readOptionsPb, final Key... keys) { + Iterator get(Optional readOptionsPb, final Key... keys) { if (keys.length == 0) { return Collections.emptyIterator(); } com.google.datastore.v1.LookupRequest.Builder requestPb = com.google.datastore.v1.LookupRequest.newBuilder(); - if (readOptionsPb != null) { - requestPb.setReadOptions(readOptionsPb); - } + readOptionsPb.ifPresent(requestPb::setReadOptions); for (Key k : Sets.newLinkedHashSet(Arrays.asList(keys))) { requestPb.addKeys(k.toPb()); } diff --git a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/GqlQuery.java b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/GqlQuery.java index bebb4df9f..d4f9f3534 100644 --- a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/GqlQuery.java +++ b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/GqlQuery.java @@ -456,6 +456,16 @@ public Map getNamedBindings() { return builder.buildOrThrow(); } + @InternalApi + public Map getNamedBindingsMap() { + return namedBindings; + } + + @InternalApi + public List getPositionalBindingsMap() { + return positionalBindings; + } + /** Returns an immutable list of positional bindings (using original order). */ public List getNumberArgs() { ImmutableList.Builder builder = ImmutableList.builder(); @@ -504,17 +514,8 @@ public boolean equals(Object obj) { } com.google.datastore.v1.GqlQuery toPb() { - com.google.datastore.v1.GqlQuery.Builder queryPb = - com.google.datastore.v1.GqlQuery.newBuilder(); - queryPb.setQueryString(queryString); - queryPb.setAllowLiterals(allowLiteral); - for (Map.Entry entry : namedBindings.entrySet()) { - queryPb.putNamedBindings(entry.getKey(), entry.getValue().toPb()); - } - for (Binding argument : positionalBindings) { - queryPb.addPositionalBindings(argument.toPb()); - } - return queryPb.build(); + GqlQueryProtoPreparer protoPreparer = new GqlQueryProtoPreparer(); + return protoPreparer.prepare(this); } @InternalApi diff --git a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/GqlQueryProtoPreparer.java b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/GqlQueryProtoPreparer.java new file mode 100644 index 000000000..5269740f7 --- /dev/null +++ b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/GqlQueryProtoPreparer.java @@ -0,0 +1,43 @@ +/* + * Copyright 2022 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.google.cloud.datastore; + +import com.google.api.core.InternalApi; +import com.google.cloud.datastore.GqlQuery.Binding; +import com.google.cloud.datastore.execution.request.ProtoPreparer; +import java.util.Map; + +@InternalApi +public class GqlQueryProtoPreparer + implements ProtoPreparer, com.google.datastore.v1.GqlQuery> { + + @Override + public com.google.datastore.v1.GqlQuery prepare(GqlQuery gqlQuery) { + com.google.datastore.v1.GqlQuery.Builder queryPb = + com.google.datastore.v1.GqlQuery.newBuilder(); + + queryPb.setQueryString(gqlQuery.getQueryString()); + queryPb.setAllowLiterals(gqlQuery.allowLiteral()); + for (Map.Entry entry : gqlQuery.getNamedBindingsMap().entrySet()) { + queryPb.putNamedBindings(entry.getKey(), entry.getValue().toPb()); + } + for (Binding argument : gqlQuery.getPositionalBindingsMap()) { + queryPb.addPositionalBindings(argument.toPb()); + } + + return queryPb.build(); + } +} diff --git a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/QueryResultsImpl.java b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/QueryResultsImpl.java index 222efa3b8..6170c0b8b 100644 --- a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/QueryResultsImpl.java +++ b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/QueryResultsImpl.java @@ -20,14 +20,16 @@ import com.google.common.base.Preconditions; import com.google.common.collect.AbstractIterator; import com.google.datastore.v1.QueryResultBatch.MoreResultsType; +import com.google.datastore.v1.ReadOptions; import com.google.protobuf.ByteString; import java.util.Iterator; import java.util.Objects; +import java.util.Optional; class QueryResultsImpl extends AbstractIterator implements QueryResults { private final DatastoreImpl datastore; - private final com.google.datastore.v1.ReadOptions readOptionsPb; + private final Optional readOptionsPb; private final com.google.datastore.v1.PartitionId partitionIdPb; private final ResultType queryResultType; private RecordQuery query; @@ -41,7 +43,7 @@ class QueryResultsImpl extends AbstractIterator implements QueryResults QueryResultsImpl( DatastoreImpl datastore, - com.google.datastore.v1.ReadOptions readOptionsPb, + Optional readOptionsPb, RecordQuery query, String namespace) { this.datastore = datastore; @@ -68,9 +70,7 @@ class QueryResultsImpl extends AbstractIterator implements QueryResults private void sendRequest() { com.google.datastore.v1.RunQueryRequest.Builder requestPb = com.google.datastore.v1.RunQueryRequest.newBuilder(); - if (readOptionsPb != null) { - requestPb.setReadOptions(readOptionsPb); - } + readOptionsPb.ifPresent(requestPb::setReadOptions); requestPb.setPartitionId(partitionIdPb); query.populatePb(requestPb); runQueryResponsePb = datastore.runQuery(requestPb.build()); diff --git a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/ReadOption.java b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/ReadOption.java index a30533e2d..30234f1d3 100644 --- a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/ReadOption.java +++ b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/ReadOption.java @@ -17,9 +17,12 @@ package com.google.cloud.datastore; import com.google.api.core.BetaApi; +import com.google.api.core.InternalApi; import com.google.cloud.Timestamp; import com.google.common.collect.ImmutableMap; +import com.google.protobuf.ByteString; import java.io.Serializable; +import java.util.List; import java.util.Map; /** @@ -68,6 +71,21 @@ public Timestamp time() { } } + /** Specifies transaction to be used when running a {@link Query}. */ + @InternalApi + static class TransactionId extends ReadOption { + + private final ByteString transactionId; + + TransactionId(ByteString transactionId) { + this.transactionId = transactionId; + } + + public ByteString getTransactionId() { + return transactionId; + } + } + private ReadOption() {} /** @@ -88,6 +106,24 @@ public static ReadTime readTime(Timestamp time) { return new ReadTime(time); } + /** + * Returns a {@code ReadOption} that specifies transaction id, allowing Datastore to execute a + * {@link Query} in this transaction. + */ + @InternalApi + public static ReadOption transactionId(String transactionId) { + return new TransactionId(ByteString.copyFrom(transactionId.getBytes())); + } + + /** + * Returns a {@code ReadOption} that specifies transaction id, allowing Datastore to execute a + * {@link Query} in this transaction. + */ + @InternalApi + public static ReadOption transactionId(ByteString transactionId) { + return new TransactionId(transactionId); + } + static Map, ReadOption> asImmutableMap(ReadOption... options) { ImmutableMap.Builder, ReadOption> builder = ImmutableMap.builder(); for (ReadOption option : options) { @@ -95,4 +131,12 @@ static Map, ReadOption> asImmutableMap(ReadOption... } return builder.buildOrThrow(); } + + static Map, ReadOption> asImmutableMap(List options) { + ImmutableMap.Builder, ReadOption> builder = ImmutableMap.builder(); + for (ReadOption option : options) { + builder.put(option.getClass(), option); + } + return builder.buildOrThrow(); + } } diff --git a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/ReadOptionProtoPreparer.java b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/ReadOptionProtoPreparer.java new file mode 100644 index 000000000..15713b02f --- /dev/null +++ b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/ReadOptionProtoPreparer.java @@ -0,0 +1,77 @@ +/* + * Copyright 2022 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.google.cloud.datastore; + +import com.google.api.core.InternalApi; +import com.google.cloud.datastore.ReadOption.EventualConsistency; +import com.google.cloud.datastore.ReadOption.ReadTime; +import com.google.cloud.datastore.ReadOption.TransactionId; +import com.google.cloud.datastore.execution.request.ProtoPreparer; +import com.google.datastore.v1.ReadOptions; +import com.google.datastore.v1.ReadOptions.ReadConsistency; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.stream.Collectors; + +@InternalApi +public class ReadOptionProtoPreparer + implements ProtoPreparer, Optional> { + + @Override + public Optional prepare(List options) { + if (options == null || options.isEmpty()) { + return Optional.empty(); + } + com.google.datastore.v1.ReadOptions readOptionsPb = null; + Map, ReadOption> optionsByType = ReadOption.asImmutableMap(options); + + boolean moreThanOneReadOption = optionsByType.keySet().size() > 1; + if (moreThanOneReadOption) { + throw DatastoreException.throwInvalidRequest( + String.format("Can not use %s together.", getInvalidOptions(optionsByType))); + } + + if (optionsByType.containsKey(EventualConsistency.class)) { + readOptionsPb = ReadOptions.newBuilder().setReadConsistency(ReadConsistency.EVENTUAL).build(); + } + + if (optionsByType.containsKey(ReadTime.class)) { + readOptionsPb = + ReadOptions.newBuilder() + .setReadTime(((ReadTime) optionsByType.get(ReadTime.class)).time().toProto()) + .build(); + } + + if (optionsByType.containsKey(TransactionId.class)) { + readOptionsPb = + ReadOptions.newBuilder() + .setTransaction( + ((TransactionId) optionsByType.get(TransactionId.class)).getTransactionId()) + .build(); + } + return Optional.ofNullable(readOptionsPb); + } + + private String getInvalidOptions(Map, ReadOption> optionsByType) { + String regex = "([a-z])([A-Z]+)"; + String replacement = "$1 $2"; + return optionsByType.keySet().stream() + .map(Class::getSimpleName) + .map(s -> s.replaceAll(regex, replacement).toLowerCase()) + .collect(Collectors.joining(", ")); + } +} diff --git a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/StructuredQuery.java b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/StructuredQuery.java index 8d2974c15..93f70aead 100644 --- a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/StructuredQuery.java +++ b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/StructuredQuery.java @@ -1045,40 +1045,8 @@ public StructuredQuery nextQuery(com.google.datastore.v1.RunQueryResponse res } com.google.datastore.v1.Query toPb() { - com.google.datastore.v1.Query.Builder queryPb = com.google.datastore.v1.Query.newBuilder(); - if (kind != null) { - queryPb.addKindBuilder().setName(kind); - } - if (startCursor != null) { - queryPb.setStartCursor(startCursor.getByteString()); - } - if (endCursor != null) { - queryPb.setEndCursor(endCursor.getByteString()); - } - if (offset > 0) { - queryPb.setOffset(offset); - } - if (limit != null) { - queryPb.setLimit(com.google.protobuf.Int32Value.newBuilder().setValue(limit)); - } - if (filter != null) { - queryPb.setFilter(filter.toPb()); - } - for (OrderBy value : orderBy) { - queryPb.addOrder(value.toPb()); - } - for (String value : distinctOn) { - queryPb.addDistinctOn( - com.google.datastore.v1.PropertyReference.newBuilder().setName(value).build()); - } - for (String value : projection) { - com.google.datastore.v1.Projection.Builder expressionPb = - com.google.datastore.v1.Projection.newBuilder(); - expressionPb.setProperty( - com.google.datastore.v1.PropertyReference.newBuilder().setName(value).build()); - queryPb.addProjection(expressionPb.build()); - } - return queryPb.build(); + StructuredQueryProtoPreparer protoPreparer = new StructuredQueryProtoPreparer(); + return protoPreparer.prepare(this); } @SuppressWarnings("unchecked") diff --git a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/StructuredQueryProtoPreparer.java b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/StructuredQueryProtoPreparer.java new file mode 100644 index 000000000..fda6f8f4a --- /dev/null +++ b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/StructuredQueryProtoPreparer.java @@ -0,0 +1,66 @@ +/* + * Copyright 2022 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.google.cloud.datastore; + +import com.google.api.core.InternalApi; +import com.google.cloud.datastore.StructuredQuery.OrderBy; +import com.google.cloud.datastore.execution.request.ProtoPreparer; +import com.google.datastore.v1.Query; +import com.google.protobuf.Int32Value; + +@InternalApi +public class StructuredQueryProtoPreparer implements ProtoPreparer, Query> { + + @Override + public Query prepare(StructuredQuery query) { + com.google.datastore.v1.Query.Builder queryPb = com.google.datastore.v1.Query.newBuilder(); + if (query.getKind() != null) { + queryPb.addKindBuilder().setName(query.getKind()); + } + if (query.getStartCursor() != null) { + queryPb.setStartCursor(query.getStartCursor().getByteString()); + } + if (query.getEndCursor() != null) { + queryPb.setEndCursor(query.getEndCursor().getByteString()); + } + if (query.getOffset() > 0) { + queryPb.setOffset(query.getOffset()); + } + if (query.getLimit() != null) { + queryPb.setLimit(Int32Value.of(query.getLimit())); + } + if (query.getFilter() != null) { + queryPb.setFilter(query.getFilter().toPb()); + } + for (OrderBy value : query.getOrderBy()) { + queryPb.addOrder(value.toPb()); + } + for (String value : query.getDistinctOn()) { + queryPb.addDistinctOn( + com.google.datastore.v1.PropertyReference.newBuilder().setName(value).build()); + } + for (String value : query.getProjection()) { + com.google.datastore.v1.Projection expressionPb = + com.google.datastore.v1.Projection.newBuilder() + .setProperty( + com.google.datastore.v1.PropertyReference.newBuilder().setName(value).build()) + .build(); + queryPb.addProjection(expressionPb); + } + + return queryPb.build(); + } +} diff --git a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/TransactionImpl.java b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/TransactionImpl.java index 3318ec866..94edc2216 100644 --- a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/TransactionImpl.java +++ b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/TransactionImpl.java @@ -16,11 +16,14 @@ package com.google.cloud.datastore; +import com.google.common.collect.ImmutableList; +import com.google.datastore.v1.ReadOptions; import com.google.datastore.v1.TransactionOptions; import com.google.protobuf.ByteString; import java.util.ArrayList; import java.util.Iterator; import java.util.List; +import java.util.Optional; final class TransactionImpl extends BaseDatastoreBatchWriter implements Transaction { @@ -28,6 +31,8 @@ final class TransactionImpl extends BaseDatastoreBatchWriter implements Transact private final ByteString transactionId; private boolean rolledback; + private final ReadOptionProtoPreparer readOptionProtoPreparer; + static class ResponseImpl implements Transaction.Response { private final com.google.datastore.v1.CommitResponse response; @@ -65,6 +70,7 @@ public List getGeneratedKeys() { } transactionId = datastore.requestTransactionId(requestPb); + this.readOptionProtoPreparer = new ReadOptionProtoPreparer(); } @Override @@ -75,10 +81,10 @@ public Entity get(Key key) { @Override public Iterator get(Key... keys) { validateActive(); - com.google.datastore.v1.ReadOptions.Builder readOptionsPb = - com.google.datastore.v1.ReadOptions.newBuilder(); - readOptionsPb.setTransaction(transactionId); - return datastore.get(readOptionsPb.build(), keys); + Optional readOptions = + this.readOptionProtoPreparer.prepare( + ImmutableList.of(ReadOption.transactionId(transactionId))); + return datastore.get(readOptions, keys); } @Override @@ -90,10 +96,10 @@ public List fetch(Key... keys) { @Override public QueryResults run(Query query) { validateActive(); - com.google.datastore.v1.ReadOptions.Builder readOptionsPb = - com.google.datastore.v1.ReadOptions.newBuilder(); - readOptionsPb.setTransaction(transactionId); - return datastore.run(readOptionsPb.build(), query); + Optional readOptions = + this.readOptionProtoPreparer.prepare( + ImmutableList.of(ReadOption.transactionId(transactionId))); + return datastore.run(readOptions, query); } @Override diff --git a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/execution/request/ProtoPreparer.java b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/execution/request/ProtoPreparer.java new file mode 100644 index 000000000..270169965 --- /dev/null +++ b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/execution/request/ProtoPreparer.java @@ -0,0 +1,30 @@ +/* + * Copyright 2022 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.google.cloud.datastore.execution.request; + +import com.google.api.core.InternalApi; + +/** + * An internal functional interface whose implementation has the responsibility to populate a Proto + * object from a domain object. + * + * @param the type of domain object. + * @param the type of proto object + */ +@InternalApi +public interface ProtoPreparer { + OUTPUT prepare(INPUT input); +} diff --git a/google-cloud-datastore/src/test/java/com/google/cloud/datastore/GqlQueryProtoPreparerTest.java b/google-cloud-datastore/src/test/java/com/google/cloud/datastore/GqlQueryProtoPreparerTest.java new file mode 100644 index 000000000..0d2e0ede7 --- /dev/null +++ b/google-cloud-datastore/src/test/java/com/google/cloud/datastore/GqlQueryProtoPreparerTest.java @@ -0,0 +1,74 @@ +/* + * Copyright 2022 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.google.cloud.datastore; + +import static com.google.cloud.datastore.ProtoTestData.gqlQueryParameter; +import static com.google.cloud.datastore.ProtoTestData.intValue; +import static com.google.cloud.datastore.ProtoTestData.stringValue; +import static com.google.cloud.datastore.Query.newGqlQueryBuilder; +import static com.google.common.truth.Truth.assertThat; +import static java.util.Arrays.asList; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; + +import com.google.common.collect.ImmutableMap; +import java.util.HashMap; +import org.junit.Test; + +public class GqlQueryProtoPreparerTest { + + private final GqlQueryProtoPreparer protoPreparer = new GqlQueryProtoPreparer(); + private final GqlQuery.Builder gqlQueryBuilder = newGqlQueryBuilder("SELECT * from Character"); + + @Test + public void testQueryString() { + com.google.datastore.v1.GqlQuery gqlQuery = protoPreparer.prepare(gqlQueryBuilder.build()); + + assertThat(gqlQuery.getQueryString()).isEqualTo("SELECT * from Character"); + } + + @Test + public void testAllowLiteral() { + assertTrue( + protoPreparer.prepare(gqlQueryBuilder.setAllowLiteral(true).build()).getAllowLiterals()); + assertFalse( + protoPreparer.prepare(gqlQueryBuilder.setAllowLiteral(false).build()).getAllowLiterals()); + } + + @Test + public void testNamedBinding() { + com.google.datastore.v1.GqlQuery gqlQuery = + protoPreparer.prepare( + gqlQueryBuilder.setBinding("name", "John Doe").setBinding("age", 27).build()); + + assertThat(gqlQuery.getNamedBindingsMap()) + .isEqualTo( + new HashMap<>( + ImmutableMap.of( + "name", gqlQueryParameter(stringValue("John Doe")), + "age", gqlQueryParameter(intValue(27))))); + } + + @Test + public void testPositionalBinding() { + com.google.datastore.v1.GqlQuery gqlQuery = + protoPreparer.prepare(gqlQueryBuilder.addBinding("John Doe").addBinding(27).build()); + + assertThat(gqlQuery.getPositionalBindingsList()) + .isEqualTo( + asList(gqlQueryParameter(stringValue("John Doe")), gqlQueryParameter(intValue(27)))); + } +} diff --git a/google-cloud-datastore/src/test/java/com/google/cloud/datastore/ProtoTestData.java b/google-cloud-datastore/src/test/java/com/google/cloud/datastore/ProtoTestData.java new file mode 100644 index 000000000..a923b618b --- /dev/null +++ b/google-cloud-datastore/src/test/java/com/google/cloud/datastore/ProtoTestData.java @@ -0,0 +1,76 @@ +/* + * Copyright 2022 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.google.cloud.datastore; + +import static com.google.datastore.v1.PropertyOrder.Direction.ASCENDING; + +import com.google.datastore.v1.Filter; +import com.google.datastore.v1.GqlQueryParameter; +import com.google.datastore.v1.KindExpression; +import com.google.datastore.v1.Projection; +import com.google.datastore.v1.PropertyFilter.Operator; +import com.google.datastore.v1.PropertyOrder; +import com.google.datastore.v1.PropertyReference; +import com.google.datastore.v1.Value; + +public class ProtoTestData { + + public static Value booleanValue(boolean value) { + return Value.newBuilder().setBooleanValue(value).build(); + } + + public static Value stringValue(String value) { + return Value.newBuilder().setStringValue(value).build(); + } + + public static Value intValue(long value) { + return Value.newBuilder().setIntegerValue(value).build(); + } + + public static GqlQueryParameter gqlQueryParameter(Value value) { + return GqlQueryParameter.newBuilder().setValue(value).build(); + } + + public static KindExpression kind(String kind) { + return KindExpression.newBuilder().setName(kind).build(); + } + + public static Filter propertyFilter(String propertyName, Operator operator, Value value) { + return Filter.newBuilder() + .setPropertyFilter( + com.google.datastore.v1.PropertyFilter.newBuilder() + .setProperty(propertyReference(propertyName)) + .setOp(operator) + .setValue(value) + .build()) + .build(); + } + + public static PropertyReference propertyReference(String value) { + return PropertyReference.newBuilder().setName(value).build(); + } + + public static PropertyOrder propertyOrder(String value) { + return PropertyOrder.newBuilder() + .setProperty(propertyReference(value)) + .setDirection(ASCENDING) + .build(); + } + + public static Projection projection(String value) { + return Projection.newBuilder().setProperty(propertyReference(value)).build(); + } +} diff --git a/google-cloud-datastore/src/test/java/com/google/cloud/datastore/ReadOptionProtoPreparerTest.java b/google-cloud-datastore/src/test/java/com/google/cloud/datastore/ReadOptionProtoPreparerTest.java new file mode 100644 index 000000000..bda5de3b5 --- /dev/null +++ b/google-cloud-datastore/src/test/java/com/google/cloud/datastore/ReadOptionProtoPreparerTest.java @@ -0,0 +1,97 @@ +/* + * Copyright 2022 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.google.cloud.datastore; + +import static com.google.cloud.datastore.ReadOption.eventualConsistency; +import static com.google.cloud.datastore.ReadOption.readTime; +import static com.google.cloud.datastore.ReadOption.transactionId; +import static com.google.common.truth.Truth.assertThat; +import static com.google.datastore.v1.ReadOptions.ReadConsistency.EVENTUAL; +import static java.util.Collections.singletonList; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertThrows; + +import com.google.cloud.Timestamp; +import com.google.common.collect.ImmutableList; +import com.google.datastore.v1.ReadOptions; +import java.util.Arrays; +import java.util.Optional; +import org.junit.Test; + +public class ReadOptionProtoPreparerTest { + + private final ReadOptionProtoPreparer protoPreparer = new ReadOptionProtoPreparer(); + + @Test + public void shouldThrowErrorWhenUsingMultipleReadOptions() { + assertThrows( + DatastoreException.class, + () -> + protoPreparer.prepare(Arrays.asList(eventualConsistency(), readTime(Timestamp.now())))); + assertThrows( + DatastoreException.class, + () -> + protoPreparer.prepare( + Arrays.asList(eventualConsistency(), transactionId("transaction-id")))); + assertThrows( + DatastoreException.class, + () -> + protoPreparer.prepare( + Arrays.asList(transactionId("transaction-id"), readTime(Timestamp.now())))); + assertThrows( + DatastoreException.class, + () -> + protoPreparer.prepare( + Arrays.asList( + eventualConsistency(), + readTime(Timestamp.now()), + transactionId("transaction-id")))); + } + + @Test + public void shouldPrepareReadOptionsWithEventualConsistency() { + Optional readOptions = protoPreparer.prepare(singletonList(eventualConsistency())); + + assertThat(readOptions.get().getReadConsistency()).isEqualTo(EVENTUAL); + } + + @Test + public void shouldPrepareReadOptionsWithReadTime() { + Timestamp timestamp = Timestamp.now(); + Optional readOptions = protoPreparer.prepare(singletonList(readTime(timestamp))); + + assertThat(Timestamp.fromProto(readOptions.get().getReadTime())).isEqualTo(timestamp); + } + + @Test + public void shouldPrepareReadOptionsWithTransactionId() { + String dummyTransactionId = "transaction-id"; + Optional readOptions = + protoPreparer.prepare(singletonList(transactionId(dummyTransactionId))); + + assertThat(readOptions.get().getTransaction().toStringUtf8()).isEqualTo(dummyTransactionId); + } + + @Test + public void shouldReturnNullWhenReadOptionsIsNull() { + assertFalse(protoPreparer.prepare(null).isPresent()); + } + + @Test + public void shouldReturnNullWhenReadOptionsIsAnEmptyList() { + assertFalse(protoPreparer.prepare(ImmutableList.of()).isPresent()); + } +} diff --git a/google-cloud-datastore/src/test/java/com/google/cloud/datastore/StructuredQueryProtoPreparerTest.java b/google-cloud-datastore/src/test/java/com/google/cloud/datastore/StructuredQueryProtoPreparerTest.java new file mode 100644 index 000000000..60937fc28 --- /dev/null +++ b/google-cloud-datastore/src/test/java/com/google/cloud/datastore/StructuredQueryProtoPreparerTest.java @@ -0,0 +1,118 @@ +/* + * Copyright 2022 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.google.cloud.datastore; + +import static com.google.cloud.datastore.ProtoTestData.booleanValue; +import static com.google.cloud.datastore.ProtoTestData.projection; +import static com.google.cloud.datastore.ProtoTestData.propertyFilter; +import static com.google.cloud.datastore.ProtoTestData.propertyOrder; +import static com.google.cloud.datastore.ProtoTestData.propertyReference; +import static com.google.cloud.datastore.Query.newEntityQueryBuilder; +import static com.google.common.truth.Truth.assertThat; +import static com.google.datastore.v1.PropertyFilter.Operator.EQUAL; + +import com.google.cloud.datastore.StructuredQuery.OrderBy; +import com.google.cloud.datastore.StructuredQuery.PropertyFilter; +import com.google.datastore.v1.KindExpression; +import com.google.datastore.v1.Query; +import com.google.protobuf.ByteString; +import com.google.protobuf.Int32Value; +import org.junit.Test; + +public class StructuredQueryProtoPreparerTest { + + private final StructuredQueryProtoPreparer protoPreparer = new StructuredQueryProtoPreparer(); + + @Test + public void testKind() { + Query queryProto = protoPreparer.prepare(newEntityQueryBuilder().setKind("kind").build()); + + assertThat(queryProto.getKind(0)) + .isEqualTo(KindExpression.newBuilder().setName("kind").build()); + } + + @Test + public void testStartCursor() { + byte[] bytes = {1, 2}; + Query queryProto = + protoPreparer.prepare( + newEntityQueryBuilder().setStartCursor(Cursor.copyFrom(bytes)).build()); + + assertThat(queryProto.getStartCursor()).isEqualTo(ByteString.copyFrom(bytes)); + } + + @Test + public void testEndCursor() { + byte[] bytes = {1, 2}; + Query queryProto = + protoPreparer.prepare(newEntityQueryBuilder().setEndCursor(Cursor.copyFrom(bytes)).build()); + + assertThat(queryProto.getEndCursor()).isEqualTo(ByteString.copyFrom(bytes)); + } + + @Test + public void testOffset() { + Query queryProto = protoPreparer.prepare(newEntityQueryBuilder().setOffset(5).build()); + + assertThat(queryProto.getOffset()).isEqualTo(5); + } + + @Test + public void testLimit() { + Query queryProto = protoPreparer.prepare(newEntityQueryBuilder().setLimit(5).build()); + + assertThat(queryProto.getLimit()).isEqualTo(Int32Value.of(5)); + } + + @Test + public void testFilter() { + Query queryProto = + protoPreparer.prepare( + newEntityQueryBuilder().setFilter(PropertyFilter.eq("done", true)).build()); + + assertThat(queryProto.getFilter()).isEqualTo(propertyFilter("done", EQUAL, booleanValue(true))); + } + + @Test + public void testOrderBy() { + Query queryProto = + protoPreparer.prepare( + newEntityQueryBuilder() + .setOrderBy(OrderBy.asc("dept-id"), OrderBy.asc("rank")) + .build()); + + assertThat(queryProto.getOrder(0)).isEqualTo(propertyOrder("dept-id")); + assertThat(queryProto.getOrder(1)).isEqualTo(propertyOrder("rank")); + } + + @Test + public void testDistinctOn() { + Query queryProto = + protoPreparer.prepare(newEntityQueryBuilder().setDistinctOn("dept-id", "rank").build()); + + assertThat(queryProto.getDistinctOn(0)).isEqualTo(propertyReference("dept-id")); + assertThat(queryProto.getDistinctOn(1)).isEqualTo(propertyReference("rank")); + } + + @Test + public void testProjections() { + Query queryProto = + protoPreparer.prepare(newEntityQueryBuilder().setProjection("dept-id", "rank").build()); + + assertThat(queryProto.getProjection(0)).isEqualTo(projection("dept-id")); + assertThat(queryProto.getProjection(1)).isEqualTo(projection("rank")); + } +} From 8c22e61f8a0307a59301259f83a16c8324fa1b6f Mon Sep 17 00:00:00 2001 From: Prateek Date: Mon, 17 Oct 2022 21:05:31 +0530 Subject: [PATCH 07/11] feat: Count API (#823) * Add method in Datastore client to invoke rpc for aggregation query * Creating count aggregation and using it to populate Aggregation proto * Moving aggregation builder method to root level aggregation class * Introducing RecordQuery to represent queries which returns entity records when executed * Updating gitignore with patch extension * Setting up structure of Aggregation query and its builder * Introducing ProtoPreparer to populate the request protos * Delegating responsibility of preparing query proto to QueryPreparer * Populating aggregation query with nested structured query * Delegating responsibility of preparing query proto in GqlQuery to QueryPreparer * Removing RecordQuery from the query hierarchy and making it a standalone interface for now * Populating aggregation query with nested gql query * Removing deprecation warning by using assertThrows instead of ExpectedException rule * Making DatastoreRpc call aggregation query method on client * Creating response transformer to transform aggregation query response into domain objects * Implementing aggregation query executor to execute AggergationQuery * Adding missing assertion statements * Creating RetryExecutor to inject it as a dependency in other components * Making RetryExecutor accept RetrySettings when creating it * Revert "Making RetryExecutor accept RetrySettings when creating it" This reverts commit 1dfafb7f8adcdd4ec566ac6fc712d2233623d33b. * Revert "Creating RetryExecutor to inject it as a dependency in other components" This reverts commit 8872a55cd1b1e22643222aa62a17049169dee71f. * Introducing RetryAndTraceDatastoreRpcDecorator to have retry and traceability logic on top of another DatastoreRpc * Extracting out the responsibility of preparing ReadOption in it's own ProtoPreparer * Making QueryExecutor to execute query with provided ReadOptions * Exposing readTime to the user * Ignoring runAggregationQuery method from clirr check * Making readTime final * Allowing namespace to be optional in AggregationQuery * Add capability to fetch aggregation result by passing alias * Implementing User facing datastore.runAggrgation method to run aggregation query * Add integration test for count aggregation * Add transaction Id support in ReadOptionsProtoPreparer * Supporting aggregation query with transactions * Allowing user to create Aggregation directly without involving its builder * Preventing creating duplicated aggregation when creating an aggregation query * Marking RecordQuery implemented method as InternalApi * Writing comments and JavaDoc for aggregation query related class * Adding a default implementation in the public interfaces to avoid compile time failures * covering a scenario to maintain consistent snapshot when executing aggregation query in a transaction * Creating emulator proxy to simulate AggregationQuery response from emulator * Integration test to execute an aggregation query in a read only transaction * Getting rid off limit operation on count aggregation as same behaviour can be achieved by using 'limit' operation on the underlying query * Removing import statement from javadoc and undo changes in .gitignore file * Using Optional instead of returning null from ReadOptionsProtoPreparer * using assertThat from Truth library * fixing unit test * Getting rid off Double braces initialization syntax * Fixing lint * Getting rid off emulator proxy and using easy mock to check the aggregationQuery triggered * Deleting a entity created locally in other test which is causing failure in other test * Deleting all keys in datastore in integration test so that new test can start fresh * Executing two read write transaction simultaneously and verifying their behaviour * Removing tests to verify serializability as it's an underlying implementation detail * Fixing lint * Adding runAggregationQuery method to reflect config so that it's available and accessible in native image through reflection * Fixing equals of CountAggregation * Fixing lint * Adding an integration test of using limit option with aggregation query * Adding BetaApi annotation to public surface to indicate that aggregation query / count is in preview * Fixing lint * Removing unused functiona and fixing javadoc * fixing variable name --- .../google/datastore/v1/client/Datastore.java | 11 + .../META-INF/native-image/reflect-config.json | 3 +- .../v1/client/DatastoreClientTest.java | 9 + .../clirr-ignored-differences.xml | 15 + .../cloud/datastore/AggregationQuery.java | 176 ++++++++ .../cloud/datastore/AggregationResult.java | 71 ++++ .../cloud/datastore/AggregationResults.java | 82 ++++ .../com/google/cloud/datastore/Datastore.java | 49 +++ .../google/cloud/datastore/DatastoreImpl.java | 23 +- .../cloud/datastore/DatastoreReader.java | 11 + .../com/google/cloud/datastore/Query.java | 42 +- .../google/cloud/datastore/ReadOption.java | 35 ++ .../RetryAndTraceDatastoreRpcDecorator.java | 124 ++++++ .../cloud/datastore/StructuredQuery.java | 3 + .../com/google/cloud/datastore/TraceUtil.java | 2 + .../cloud/datastore/TransactionImpl.java | 13 +- .../datastore/aggregation/Aggregation.java | 47 +++ .../aggregation/AggregationBuilder.java | 31 ++ .../aggregation/CountAggregation.java | 83 ++++ .../execution/AggregationQueryExecutor.java | 66 +++ .../datastore/execution/QueryExecutor.java | 40 ++ .../AggregationQueryRequestProtoPreparer.java | 100 +++++ .../AggregationQueryResponseTransformer.java | 58 +++ .../response/ResponseTransformer.java | 30 ++ .../cloud/datastore/spi/v1/DatastoreRpc.java | 11 + .../datastore/spi/v1/HttpDatastoreRpc.java | 11 + .../cloud/datastore/AggregationQueryTest.java | 153 +++++++ .../datastore/AggregationResultTest.java | 36 ++ .../google/cloud/datastore/DatastoreTest.java | 57 +++ .../google/cloud/datastore/ProtoTestData.java | 6 + .../ReadOptionProtoPreparerTest.java | 6 +- ...etryAndTraceDatastoreRpcDecoratorTest.java | 84 ++++ .../com/google/cloud/datastore/TestUtils.java | 39 ++ .../aggregation/CountAggregationTest.java | 68 ++++ .../AggregationQueryExecutorTest.java | 177 ++++++++ ...regationQueryRequestProtoPreparerTest.java | 179 +++++++++ ...gregationQueryResponseTransformerTest.java | 91 +++++ .../cloud/datastore/it/ITDatastoreTest.java | 380 +++++++++++++++++- 38 files changed, 2410 insertions(+), 12 deletions(-) create mode 100644 google-cloud-datastore/src/main/java/com/google/cloud/datastore/AggregationQuery.java create mode 100644 google-cloud-datastore/src/main/java/com/google/cloud/datastore/AggregationResult.java create mode 100644 google-cloud-datastore/src/main/java/com/google/cloud/datastore/AggregationResults.java create mode 100644 google-cloud-datastore/src/main/java/com/google/cloud/datastore/RetryAndTraceDatastoreRpcDecorator.java create mode 100644 google-cloud-datastore/src/main/java/com/google/cloud/datastore/aggregation/Aggregation.java create mode 100644 google-cloud-datastore/src/main/java/com/google/cloud/datastore/aggregation/AggregationBuilder.java create mode 100644 google-cloud-datastore/src/main/java/com/google/cloud/datastore/aggregation/CountAggregation.java create mode 100644 google-cloud-datastore/src/main/java/com/google/cloud/datastore/execution/AggregationQueryExecutor.java create mode 100644 google-cloud-datastore/src/main/java/com/google/cloud/datastore/execution/QueryExecutor.java create mode 100644 google-cloud-datastore/src/main/java/com/google/cloud/datastore/execution/request/AggregationQueryRequestProtoPreparer.java create mode 100644 google-cloud-datastore/src/main/java/com/google/cloud/datastore/execution/response/AggregationQueryResponseTransformer.java create mode 100644 google-cloud-datastore/src/main/java/com/google/cloud/datastore/execution/response/ResponseTransformer.java create mode 100644 google-cloud-datastore/src/test/java/com/google/cloud/datastore/AggregationQueryTest.java create mode 100644 google-cloud-datastore/src/test/java/com/google/cloud/datastore/AggregationResultTest.java create mode 100644 google-cloud-datastore/src/test/java/com/google/cloud/datastore/RetryAndTraceDatastoreRpcDecoratorTest.java create mode 100644 google-cloud-datastore/src/test/java/com/google/cloud/datastore/TestUtils.java create mode 100644 google-cloud-datastore/src/test/java/com/google/cloud/datastore/aggregation/CountAggregationTest.java create mode 100644 google-cloud-datastore/src/test/java/com/google/cloud/datastore/execution/AggregationQueryExecutorTest.java create mode 100644 google-cloud-datastore/src/test/java/com/google/cloud/datastore/execution/request/AggregationQueryRequestProtoPreparerTest.java create mode 100644 google-cloud-datastore/src/test/java/com/google/cloud/datastore/execution/response/AggregationQueryResponseTransformerTest.java diff --git a/datastore-v1-proto-client/src/main/java/com/google/datastore/v1/client/Datastore.java b/datastore-v1-proto-client/src/main/java/com/google/datastore/v1/client/Datastore.java index db117142f..09101c94b 100644 --- a/datastore-v1-proto-client/src/main/java/com/google/datastore/v1/client/Datastore.java +++ b/datastore-v1-proto-client/src/main/java/com/google/datastore/v1/client/Datastore.java @@ -27,6 +27,8 @@ import com.google.datastore.v1.ReserveIdsResponse; import com.google.datastore.v1.RollbackRequest; import com.google.datastore.v1.RollbackResponse; +import com.google.datastore.v1.RunAggregationQueryRequest; +import com.google.datastore.v1.RunAggregationQueryResponse; import com.google.datastore.v1.RunQueryRequest; import com.google.datastore.v1.RunQueryResponse; import com.google.rpc.Code; @@ -120,4 +122,13 @@ public RunQueryResponse runQuery(RunQueryRequest request) throws DatastoreExcept throw invalidResponseException("runQuery", exception); } } + + public RunAggregationQueryResponse runAggregationQuery(RunAggregationQueryRequest request) + throws DatastoreException { + try (InputStream is = remoteRpc.call("runAggregationQuery", request)) { + return RunAggregationQueryResponse.parseFrom(is); + } catch (IOException exception) { + throw invalidResponseException("runAggregationQuery", exception); + } + } } diff --git a/datastore-v1-proto-client/src/main/resources/META-INF/native-image/reflect-config.json b/datastore-v1-proto-client/src/main/resources/META-INF/native-image/reflect-config.json index 32b27f5d9..17876ff43 100644 --- a/datastore-v1-proto-client/src/main/resources/META-INF/native-image/reflect-config.json +++ b/datastore-v1-proto-client/src/main/resources/META-INF/native-image/reflect-config.json @@ -8,7 +8,8 @@ {"name":"lookup","parameterTypes":["com.google.datastore.v1.LookupRequest"] }, {"name":"reserveIds","parameterTypes":["com.google.datastore.v1.ReserveIdsRequest"] }, {"name":"rollback","parameterTypes":["com.google.datastore.v1.RollbackRequest"] }, - {"name":"runQuery","parameterTypes":["com.google.datastore.v1.RunQueryRequest"] } + {"name":"runQuery","parameterTypes":["com.google.datastore.v1.RunQueryRequest"] }, + {"name":"runAggregationQuery","parameterTypes":["com.google.datastore.v1.RunAggregationQueryRequest"] } ] }, { diff --git a/datastore-v1-proto-client/src/test/java/com/google/datastore/v1/client/DatastoreClientTest.java b/datastore-v1-proto-client/src/test/java/com/google/datastore/v1/client/DatastoreClientTest.java index 2ab2c89f8..16a6303bb 100644 --- a/datastore-v1-proto-client/src/test/java/com/google/datastore/v1/client/DatastoreClientTest.java +++ b/datastore-v1-proto-client/src/test/java/com/google/datastore/v1/client/DatastoreClientTest.java @@ -38,6 +38,8 @@ import com.google.datastore.v1.ReserveIdsResponse; import com.google.datastore.v1.RollbackRequest; import com.google.datastore.v1.RollbackResponse; +import com.google.datastore.v1.RunAggregationQueryRequest; +import com.google.datastore.v1.RunAggregationQueryResponse; import com.google.datastore.v1.RunQueryRequest; import com.google.datastore.v1.RunQueryResponse; import com.google.datastore.v1.client.testing.MockCredential; @@ -336,6 +338,13 @@ public void runQuery() throws Exception { expectRpc("runQuery", request.build(), response.build()); } + @Test + public void runAggregationQuery() throws Exception { + RunAggregationQueryRequest.Builder request = RunAggregationQueryRequest.newBuilder(); + RunAggregationQueryResponse.Builder response = RunAggregationQueryResponse.newBuilder(); + expectRpc("runAggregationQuery", request.build(), response.build()); + } + private void expectRpc(String methodName, Message request, Message response) throws Exception { Datastore datastore = factory.create(options.build()); MockDatastoreFactory mockClient = (MockDatastoreFactory) factory; diff --git a/google-cloud-datastore/clirr-ignored-differences.xml b/google-cloud-datastore/clirr-ignored-differences.xml index 110f22f73..018afb17e 100644 --- a/google-cloud-datastore/clirr-ignored-differences.xml +++ b/google-cloud-datastore/clirr-ignored-differences.xml @@ -11,4 +11,19 @@ com.google.datastore.v1.ReserveIdsResponse reserveIds(com.google.datastore.v1.ReserveIdsRequest) 7012 + + com/google/cloud/datastore/spi/v1/DatastoreRpc + com.google.datastore.v1.RunAggregationQueryResponse runAggregationQuery(com.google.datastore.v1.RunAggregationQueryRequest) + 7012 + + + com/google/cloud/datastore/Datastore + com.google.cloud.datastore.AggregationResults runAggregation(com.google.cloud.datastore.AggregationQuery, com.google.cloud.datastore.ReadOption[]) + 7012 + + + com/google/cloud/datastore/DatastoreReader + com.google.cloud.datastore.AggregationResults runAggregation(com.google.cloud.datastore.AggregationQuery) + 7012 + diff --git a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/AggregationQuery.java b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/AggregationQuery.java new file mode 100644 index 000000000..05f48a6c6 --- /dev/null +++ b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/AggregationQuery.java @@ -0,0 +1,176 @@ +/* + * Copyright 2022 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.google.cloud.datastore; + +import static com.google.cloud.datastore.AggregationQuery.Mode.GQL; +import static com.google.cloud.datastore.AggregationQuery.Mode.STRUCTURED; +import static com.google.common.base.Preconditions.checkArgument; + +import com.google.api.core.BetaApi; +import com.google.cloud.datastore.aggregation.Aggregation; +import com.google.cloud.datastore.aggregation.AggregationBuilder; +import java.util.HashSet; +import java.util.Set; + +/** + * An implementation of a Google Cloud Datastore Query that returns {@link AggregationResults}, It + * can be constructed by providing a nested query ({@link StructuredQuery} or {@link GqlQuery}) to + * run the aggregations on and a set of {@link Aggregation}. + * + *

{@link StructuredQuery} example: + * + *

{@code
+ * EntityQuery selectAllQuery = Query.newEntityQueryBuilder()
+ *    .setKind("Task")
+ *    .build();
+ * AggregationQuery aggregationQuery = Query.newAggregationQueryBuilder()
+ *    .addAggregation(count().as("total_count"))
+ *    .over(selectAllQuery)
+ *    .build();
+ * AggregationResults aggregationResults = datastore.runAggregation(aggregationQuery);
+ * for (AggregationResult aggregationResult : aggregationResults) {
+ *     System.out.println(aggregationResult.get("total_count"));
+ * }
+ * }
+ * + *

{@link GqlQuery} example:

+ * + *
{@code
+ * GqlQuery selectAllGqlQuery = Query.newGqlQueryBuilder(
+ *         "AGGREGATE COUNT(*) AS total_count, COUNT_UP_TO(100) AS count_upto_100 OVER(SELECT * FROM Task)"
+ *     )
+ *     .setAllowLiteral(true)
+ *     .build();
+ * AggregationQuery aggregationQuery = Query.newAggregationQueryBuilder()
+ *     .over(selectAllGqlQuery)
+ *     .build();
+ * AggregationResults aggregationResults = datastore.runAggregation(aggregationQuery);
+ * for (AggregationResult aggregationResult : aggregationResults) {
+ *   System.out.println(aggregationResult.get("total_count"));
+ *   System.out.println(aggregationResult.get("count_upto_100"));
+ * }
+ * }
+ * + * @see Datastore + * queries + */ +@BetaApi +public class AggregationQuery extends Query { + + private Set aggregations; + private StructuredQuery nestedStructuredQuery; + private final Mode mode; + private GqlQuery nestedGqlQuery; + + AggregationQuery( + String namespace, Set aggregations, StructuredQuery nestedQuery) { + super(namespace); + checkArgument( + !aggregations.isEmpty(), + "At least one aggregation is required for an aggregation query to run"); + this.aggregations = aggregations; + this.nestedStructuredQuery = nestedQuery; + this.mode = STRUCTURED; + } + + AggregationQuery(String namespace, GqlQuery gqlQuery) { + super(namespace); + this.nestedGqlQuery = gqlQuery; + this.mode = GQL; + } + + /** Returns the {@link Aggregation}(s) for this Query. */ + public Set getAggregations() { + return aggregations; + } + + /** + * Returns the underlying {@link StructuredQuery for this Query}. Returns null if created with + * {@link GqlQuery} + */ + public StructuredQuery getNestedStructuredQuery() { + return nestedStructuredQuery; + } + + /** + * Returns the underlying {@link GqlQuery for this Query}. Returns null if created with {@link + * StructuredQuery} + */ + public GqlQuery getNestedGqlQuery() { + return nestedGqlQuery; + } + + /** Returns the {@link Mode} for this query. */ + public Mode getMode() { + return mode; + } + + public static class Builder { + + private String namespace; + private Mode mode; + private final Set aggregations; + private StructuredQuery nestedStructuredQuery; + private GqlQuery nestedGqlQuery; + + public Builder() { + this.aggregations = new HashSet<>(); + } + + public Builder setNamespace(String namespace) { + this.namespace = namespace; + return this; + } + + public Builder addAggregation(AggregationBuilder aggregationBuilder) { + this.aggregations.add(aggregationBuilder.build()); + return this; + } + + public Builder addAggregation(Aggregation aggregation) { + this.aggregations.add(aggregation); + return this; + } + + public Builder over(StructuredQuery nestedQuery) { + this.nestedStructuredQuery = nestedQuery; + this.mode = STRUCTURED; + return this; + } + + public Builder over(GqlQuery nestedQuery) { + this.nestedGqlQuery = nestedQuery; + this.mode = GQL; + return this; + } + + public AggregationQuery build() { + boolean nestedQueryProvided = nestedGqlQuery != null || nestedStructuredQuery != null; + checkArgument( + nestedQueryProvided, "Nested query is required for an aggregation query to run"); + + if (mode == GQL) { + return new AggregationQuery(namespace, nestedGqlQuery); + } + return new AggregationQuery(namespace, aggregations, nestedStructuredQuery); + } + } + + public enum Mode { + STRUCTURED, + GQL, + } +} diff --git a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/AggregationResult.java b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/AggregationResult.java new file mode 100644 index 000000000..928997ee7 --- /dev/null +++ b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/AggregationResult.java @@ -0,0 +1,71 @@ +/* + * Copyright 2022 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.google.cloud.datastore; + +import com.google.api.core.BetaApi; +import com.google.common.base.MoreObjects; +import com.google.common.base.MoreObjects.ToStringHelper; +import java.util.Map; +import java.util.Map.Entry; +import java.util.Objects; + +/** Represents a result of an {@link AggregationQuery} query submission. */ +@BetaApi +public class AggregationResult { + + private final Map properties; + + public AggregationResult(Map properties) { + this.properties = properties; + } + + /** + * Returns a result value for the given alias. + * + * @param alias A custom alias provided in the query or an autogenerated alias in the form of + * 'property_\d' + * @return An aggregation result value for the given alias. + */ + public Long get(String alias) { + return properties.get(alias).get(); + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (o == null || getClass() != o.getClass()) { + return false; + } + AggregationResult that = (AggregationResult) o; + return properties.equals(that.properties); + } + + @Override + public int hashCode() { + return Objects.hash(properties); + } + + @Override + public String toString() { + ToStringHelper toStringHelper = MoreObjects.toStringHelper(this); + for (Entry entry : properties.entrySet()) { + toStringHelper.add(entry.getKey(), entry.getValue().get()); + } + return toStringHelper.toString(); + } +} diff --git a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/AggregationResults.java b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/AggregationResults.java new file mode 100644 index 000000000..feff5b805 --- /dev/null +++ b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/AggregationResults.java @@ -0,0 +1,82 @@ +/* + * Copyright 2022 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.google.cloud.datastore; + +import static com.google.api.client.util.Preconditions.checkNotNull; + +import com.google.api.core.BetaApi; +import com.google.api.core.InternalApi; +import com.google.cloud.Timestamp; +import java.util.Iterator; +import java.util.List; +import java.util.Objects; + +/** + * The result of an {@link AggregationQuery} query submission. Contains a {@link + * List} and readTime {@link Timestamp} in it. + * + *

This can be used to iterate over an underlying {@link List} directly. + */ +@BetaApi +public class AggregationResults implements Iterable { + + private final List aggregationResults; + private final Timestamp readTime; + + public AggregationResults(List aggregationResults, Timestamp readTime) { + checkNotNull(aggregationResults, "Aggregation results cannot be null"); + checkNotNull(readTime, "readTime cannot be null"); + this.aggregationResults = aggregationResults; + this.readTime = readTime; + } + + /** Returns {@link Iterator} for underlying {@link List}. */ + @Override + public Iterator iterator() { + return this.aggregationResults.iterator(); + } + + public int size() { + return this.aggregationResults.size(); + } + + @InternalApi + public AggregationResult get(int index) { + return this.aggregationResults.get(index); + } + + /** Returns read timestamp this result batch was returned from. */ + public Timestamp getReadTime() { + return this.readTime; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (o == null || getClass() != o.getClass()) { + return false; + } + AggregationResults that = (AggregationResults) o; + return Objects.equals(aggregationResults, that.aggregationResults); + } + + @Override + public int hashCode() { + return Objects.hash(aggregationResults); + } +} diff --git a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/Datastore.java b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/Datastore.java index bb115995e..9d0a21b8d 100644 --- a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/Datastore.java +++ b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/Datastore.java @@ -16,6 +16,7 @@ package com.google.cloud.datastore; +import com.google.api.core.BetaApi; import com.google.cloud.Service; import com.google.datastore.v1.TransactionOptions; import java.util.Iterator; @@ -461,4 +462,52 @@ interface TransactionCallable { * @throws DatastoreException upon failure */ QueryResults run(Query query, ReadOption... options); + + /** + * Submits a {@link AggregationQuery} and returns {@link AggregationResults}. {@link ReadOption}s + * can be specified if desired. + * + *

Example of running an {@link AggregationQuery} to find the count of entities of one kind. + * + *

{@link StructuredQuery} example: + * + *

{@code
+   * EntityQuery selectAllQuery = Query.newEntityQueryBuilder()
+   *    .setKind("Task")
+   *    .build();
+   * AggregationQuery aggregationQuery = Query.newAggregationQueryBuilder()
+   *    .addAggregation(count().as("total_count"))
+   *    .over(selectAllQuery)
+   *    .build();
+   * AggregationResults aggregationResults = datastore.runAggregation(aggregationQuery);
+   * for (AggregationResult aggregationResult : aggregationResults) {
+   *     System.out.println(aggregationResult.get("total_count"));
+   * }
+   * }
+ * + *

{@link GqlQuery} example:

+ * + *
{@code
+   * GqlQuery selectAllGqlQuery = Query.newGqlQueryBuilder(
+   *         "AGGREGATE COUNT(*) AS total_count, COUNT_UP_TO(100) AS count_upto_100 OVER(SELECT * FROM Task)"
+   *     )
+   *     .setAllowLiteral(true)
+   *     .build();
+   * AggregationQuery aggregationQuery = Query.newAggregationQueryBuilder()
+   *     .over(selectAllGqlQuery)
+   *     .build();
+   * AggregationResults aggregationResults = datastore.runAggregation(aggregationQuery);
+   * for (AggregationResult aggregationResult : aggregationResults) {
+   *   System.out.println(aggregationResult.get("total_count"));
+   *   System.out.println(aggregationResult.get("count_upto_100"));
+   * }
+   * }
+ * + * @throws DatastoreException upon failure + * @return {@link AggregationResults} + */ + @BetaApi + default AggregationResults runAggregation(AggregationQuery query, ReadOption... options) { + throw new UnsupportedOperationException("Not implemented."); + } } diff --git a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/DatastoreImpl.java b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/DatastoreImpl.java index f6a96c2a4..4f6533eca 100644 --- a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/DatastoreImpl.java +++ b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/DatastoreImpl.java @@ -16,12 +16,14 @@ package com.google.cloud.datastore; +import com.google.api.core.BetaApi; import com.google.api.gax.retrying.RetrySettings; import com.google.cloud.BaseService; import com.google.cloud.ExceptionHandler; import com.google.cloud.RetryHelper; import com.google.cloud.RetryHelper.RetryHelperException; import com.google.cloud.ServiceOptions; +import com.google.cloud.datastore.execution.AggregationQueryExecutor; import com.google.cloud.datastore.spi.v1.DatastoreRpc; import com.google.common.base.MoreObjects; import com.google.common.base.Preconditions; @@ -59,13 +61,19 @@ final class DatastoreImpl extends BaseService implements Datas private final TraceUtil traceUtil = TraceUtil.getInstance(); private final ReadOptionProtoPreparer readOptionProtoPreparer; + private final AggregationQueryExecutor aggregationQueryExecutor; DatastoreImpl(DatastoreOptions options) { super(options); this.datastoreRpc = options.getDatastoreRpcV1(); retrySettings = MoreObjects.firstNonNull(options.getRetrySettings(), ServiceOptions.getNoRetrySettings()); + readOptionProtoPreparer = new ReadOptionProtoPreparer(); + aggregationQueryExecutor = + new AggregationQueryExecutor( + new RetryAndTraceDatastoreRpcDecorator(datastoreRpc, traceUtil, retrySettings, options), + options); } @Override @@ -84,6 +92,7 @@ public Transaction newTransaction() { } static class ReadWriteTransactionCallable implements Callable { + private final Datastore datastore; private final TransactionCallable callable; private volatile TransactionOptions options; @@ -184,10 +193,22 @@ public QueryResults run(Query query, ReadOption... options) { @SuppressWarnings("unchecked") QueryResults run(Optional readOptionsPb, Query query) { - return new QueryResultsImpl<>( + return new QueryResultsImpl( this, readOptionsPb, (RecordQuery) query, query.getNamespace()); } + @Override + @BetaApi + public AggregationResults runAggregation(AggregationQuery query) { + return aggregationQueryExecutor.execute(query); + } + + @Override + @BetaApi + public AggregationResults runAggregation(AggregationQuery query, ReadOption... options) { + return aggregationQueryExecutor.execute(query, options); + } + com.google.datastore.v1.RunQueryResponse runQuery( final com.google.datastore.v1.RunQueryRequest requestPb) { Span span = traceUtil.startSpan(TraceUtil.SPAN_NAME_RUNQUERY); diff --git a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/DatastoreReader.java b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/DatastoreReader.java index 3d5b7cd3e..751f99566 100644 --- a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/DatastoreReader.java +++ b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/DatastoreReader.java @@ -16,6 +16,7 @@ package com.google.cloud.datastore; +import com.google.api.core.BetaApi; import java.util.Iterator; import java.util.List; @@ -53,4 +54,14 @@ public interface DatastoreReader { * @throws DatastoreException upon failure */ QueryResults run(Query query); + + /** + * Submits a {@link AggregationQuery} and returns {@link AggregationResults}. + * + * @throws DatastoreException upon failure + */ + @BetaApi + default AggregationResults runAggregation(AggregationQuery query) { + throw new UnsupportedOperationException("Not implemented."); + } } diff --git a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/Query.java b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/Query.java index a0bed5984..8870cf520 100644 --- a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/Query.java +++ b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/Query.java @@ -23,8 +23,8 @@ import java.util.Map; /** - * A Google Cloud Datastore query. For usage examples see {@link GqlQuery} and {@link - * StructuredQuery}. + * A Google Cloud Datastore query. For usage examples see {@link GqlQuery}, {@link StructuredQuery} + * and {@link AggregationQuery}. * *

Note that queries require proper indexing. See Cloud Datastore Index @@ -254,4 +254,42 @@ public static KeyQuery.Builder newKeyQueryBuilder() { public static ProjectionEntityQuery.Builder newProjectionEntityQueryBuilder() { return new ProjectionEntityQuery.Builder(); } + + /** + * Returns a new {@link AggregationQuery} builder. + * + *

Example of creating and running an {@link AggregationQuery}. + * + *

{@link StructuredQuery} example: + * + *

{@code
+   * EntityQuery selectAllQuery = Query.newEntityQueryBuilder()
+   *    .setKind("Task")
+   *    .build();
+   * AggregationQuery aggregationQuery = Query.newAggregationQueryBuilder()
+   *    .addAggregation(count().as("total_count"))
+   *    .over(selectAllQuery)
+   *    .build();
+   * AggregationResults aggregationResults = datastore.runAggregation(aggregationQuery);
+   * // Use aggregationResults
+   * }
+ * + *

{@link GqlQuery} example:

+ * + *
{@code
+   * GqlQuery selectAllGqlQuery = Query.newGqlQueryBuilder(
+   *         "AGGREGATE COUNT(*) AS total_count OVER(SELECT * FROM Task)"
+   *     )
+   *     .setAllowLiteral(true)
+   *     .build();
+   * AggregationQuery aggregationQuery = Query.newAggregationQueryBuilder()
+   *     .over(selectAllGqlQuery)
+   *     .build();
+   * AggregationResults aggregationResults = datastore.runAggregation(aggregationQuery);
+   * // Use aggregationResults
+   * }
+ */ + public static AggregationQuery.Builder newAggregationQueryBuilder() { + return new AggregationQuery.Builder(); + } } diff --git a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/ReadOption.java b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/ReadOption.java index 30234f1d3..be5644da0 100644 --- a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/ReadOption.java +++ b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/ReadOption.java @@ -22,6 +22,7 @@ import com.google.common.collect.ImmutableMap; import com.google.protobuf.ByteString; import java.io.Serializable; +import java.util.Collections; import java.util.List; import java.util.Map; @@ -139,4 +140,38 @@ static Map, ReadOption> asImmutableMap(List> { + + Q query; + List readOptions; + + private QueryAndReadOptions(Q query, List readOptions) { + this.query = query; + this.readOptions = readOptions; + } + + private QueryAndReadOptions(Q query) { + this.query = query; + this.readOptions = Collections.emptyList(); + } + + public Q getQuery() { + return query; + } + + public List getReadOptions() { + return readOptions; + } + + public static > QueryAndReadOptions create(Q query) { + return new QueryAndReadOptions<>(query); + } + + public static > QueryAndReadOptions create( + Q query, List readOptions) { + return new QueryAndReadOptions<>(query, readOptions); + } + } } diff --git a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/RetryAndTraceDatastoreRpcDecorator.java b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/RetryAndTraceDatastoreRpcDecorator.java new file mode 100644 index 000000000..c4a85caab --- /dev/null +++ b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/RetryAndTraceDatastoreRpcDecorator.java @@ -0,0 +1,124 @@ +/* + * Copyright 2022 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.google.cloud.datastore; + +import static com.google.cloud.BaseService.EXCEPTION_HANDLER; +import static com.google.cloud.datastore.TraceUtil.SPAN_NAME_RUN_AGGREGATION_QUERY; + +import com.google.api.core.InternalApi; +import com.google.api.gax.retrying.RetrySettings; +import com.google.cloud.RetryHelper; +import com.google.cloud.RetryHelper.RetryHelperException; +import com.google.cloud.datastore.spi.v1.DatastoreRpc; +import com.google.datastore.v1.AllocateIdsRequest; +import com.google.datastore.v1.AllocateIdsResponse; +import com.google.datastore.v1.BeginTransactionRequest; +import com.google.datastore.v1.BeginTransactionResponse; +import com.google.datastore.v1.CommitRequest; +import com.google.datastore.v1.CommitResponse; +import com.google.datastore.v1.LookupRequest; +import com.google.datastore.v1.LookupResponse; +import com.google.datastore.v1.ReserveIdsRequest; +import com.google.datastore.v1.ReserveIdsResponse; +import com.google.datastore.v1.RollbackRequest; +import com.google.datastore.v1.RollbackResponse; +import com.google.datastore.v1.RunAggregationQueryRequest; +import com.google.datastore.v1.RunAggregationQueryResponse; +import com.google.datastore.v1.RunQueryRequest; +import com.google.datastore.v1.RunQueryResponse; +import io.opencensus.common.Scope; +import io.opencensus.trace.Span; +import io.opencensus.trace.Status; +import java.util.concurrent.Callable; + +/** + * An implementation of {@link DatastoreRpc} which acts as a Decorator and decorates the underlying + * {@link DatastoreRpc} with the logic of retry and Traceability. + */ +@InternalApi +public class RetryAndTraceDatastoreRpcDecorator implements DatastoreRpc { + + private final DatastoreRpc datastoreRpc; + private final TraceUtil traceUtil; + private final RetrySettings retrySettings; + private final DatastoreOptions datastoreOptions; + + public RetryAndTraceDatastoreRpcDecorator( + DatastoreRpc datastoreRpc, + TraceUtil traceUtil, + RetrySettings retrySettings, + DatastoreOptions datastoreOptions) { + this.datastoreRpc = datastoreRpc; + this.traceUtil = traceUtil; + this.retrySettings = retrySettings; + this.datastoreOptions = datastoreOptions; + } + + @Override + public AllocateIdsResponse allocateIds(AllocateIdsRequest request) { + throw new UnsupportedOperationException("Not implemented."); + } + + @Override + public BeginTransactionResponse beginTransaction(BeginTransactionRequest request) + throws DatastoreException { + throw new UnsupportedOperationException("Not implemented."); + } + + @Override + public CommitResponse commit(CommitRequest request) { + throw new UnsupportedOperationException("Not implemented."); + } + + @Override + public LookupResponse lookup(LookupRequest request) { + throw new UnsupportedOperationException("Not implemented."); + } + + @Override + public ReserveIdsResponse reserveIds(ReserveIdsRequest request) { + throw new UnsupportedOperationException("Not implemented."); + } + + @Override + public RollbackResponse rollback(RollbackRequest request) { + throw new UnsupportedOperationException("Not implemented."); + } + + @Override + public RunQueryResponse runQuery(RunQueryRequest request) { + throw new UnsupportedOperationException("Not implemented."); + } + + @Override + public RunAggregationQueryResponse runAggregationQuery(RunAggregationQueryRequest request) { + return invokeRpc( + () -> datastoreRpc.runAggregationQuery(request), SPAN_NAME_RUN_AGGREGATION_QUERY); + } + + public O invokeRpc(Callable block, String startSpan) { + Span span = traceUtil.startSpan(startSpan); + try (Scope scope = traceUtil.getTracer().withSpan(span)) { + return RetryHelper.runWithRetries( + block, this.retrySettings, EXCEPTION_HANDLER, this.datastoreOptions.getClock()); + } catch (RetryHelperException e) { + span.setStatus(Status.UNKNOWN.withDescription(e.getMessage())); + throw DatastoreException.translateAndThrow(e); + } finally { + span.end(TraceUtil.END_SPAN_OPTIONS); + } + } +} diff --git a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/StructuredQuery.java b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/StructuredQuery.java index 93f70aead..b394dcd97 100644 --- a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/StructuredQuery.java +++ b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/StructuredQuery.java @@ -30,6 +30,7 @@ import com.google.cloud.StringEnumType; import com.google.cloud.StringEnumValue; import com.google.cloud.Timestamp; +import com.google.cloud.datastore.Query.ResultType; import com.google.common.base.MoreObjects; import com.google.common.base.MoreObjects.ToStringHelper; import com.google.common.base.Preconditions; @@ -1018,6 +1019,8 @@ public Integer getLimit() { public abstract Builder toBuilder(); + @InternalApi + @Override public ResultType getType() { return resultType; } diff --git a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/TraceUtil.java b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/TraceUtil.java index 1f28b2e80..57525d15d 100644 --- a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/TraceUtil.java +++ b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/TraceUtil.java @@ -39,6 +39,8 @@ public class TraceUtil { static final String SPAN_NAME_RESERVEIDS = "CloudDatastoreOperation.reserveIds"; static final String SPAN_NAME_ROLLBACK = "CloudDatastoreOperation.rollback"; static final String SPAN_NAME_RUNQUERY = "CloudDatastoreOperation.runQuery"; + static final String SPAN_NAME_RUN_AGGREGATION_QUERY = + "CloudDatastoreOperation.runAggregationQuery"; static final EndSpanOptions END_SPAN_OPTIONS = EndSpanOptions.builder().setSampleToLocalSpanStore(true).build(); diff --git a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/TransactionImpl.java b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/TransactionImpl.java index 94edc2216..fc6c5e944 100644 --- a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/TransactionImpl.java +++ b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/TransactionImpl.java @@ -16,6 +16,8 @@ package com.google.cloud.datastore; +import static com.google.cloud.datastore.ReadOption.transactionId; + import com.google.common.collect.ImmutableList; import com.google.datastore.v1.ReadOptions; import com.google.datastore.v1.TransactionOptions; @@ -82,8 +84,7 @@ public Entity get(Key key) { public Iterator get(Key... keys) { validateActive(); Optional readOptions = - this.readOptionProtoPreparer.prepare( - ImmutableList.of(ReadOption.transactionId(transactionId))); + this.readOptionProtoPreparer.prepare(ImmutableList.of(transactionId(transactionId))); return datastore.get(readOptions, keys); } @@ -97,11 +98,15 @@ public List fetch(Key... keys) { public QueryResults run(Query query) { validateActive(); Optional readOptions = - this.readOptionProtoPreparer.prepare( - ImmutableList.of(ReadOption.transactionId(transactionId))); + this.readOptionProtoPreparer.prepare(ImmutableList.of(transactionId(transactionId))); return datastore.run(readOptions, query); } + @Override + public AggregationResults runAggregation(AggregationQuery query) { + return datastore.runAggregation(query, transactionId(transactionId)); + } + @Override public Transaction.Response commit() { validateActive(); diff --git a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/aggregation/Aggregation.java b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/aggregation/Aggregation.java new file mode 100644 index 000000000..8a8e8cc18 --- /dev/null +++ b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/aggregation/Aggregation.java @@ -0,0 +1,47 @@ +/* + * Copyright 2022 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.google.cloud.datastore.aggregation; + +import com.google.api.core.BetaApi; +import com.google.api.core.InternalApi; +import com.google.datastore.v1.AggregationQuery; + +/** + * Represents a Google Cloud Datastore Aggregation which is used with an {@link AggregationQuery}. + */ +@BetaApi +public abstract class Aggregation { + + private final String alias; + + public Aggregation(String alias) { + this.alias = alias; + } + + /** Returns the alias for this aggregation. */ + public String getAlias() { + return alias; + } + + @InternalApi + public abstract AggregationQuery.Aggregation toPb(); + + /** Returns a {@link CountAggregation} builder. */ + public static CountAggregation.Builder count() { + return new CountAggregation.Builder(); + } +} diff --git a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/aggregation/AggregationBuilder.java b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/aggregation/AggregationBuilder.java new file mode 100644 index 000000000..5e90b86aa --- /dev/null +++ b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/aggregation/AggregationBuilder.java @@ -0,0 +1,31 @@ +/* + * Copyright 2022 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.google.cloud.datastore.aggregation; + +import com.google.api.core.BetaApi; + +/** + * An interface to represent the builders which build and customize {@link Aggregation} for {@link + * com.google.cloud.datastore.AggregationQuery}. + * + *

Used by {@link + * com.google.cloud.datastore.AggregationQuery.Builder#addAggregation(AggregationBuilder)}. + */ +@BetaApi +public interface AggregationBuilder { + A build(); +} diff --git a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/aggregation/CountAggregation.java b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/aggregation/CountAggregation.java new file mode 100644 index 000000000..a5295addf --- /dev/null +++ b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/aggregation/CountAggregation.java @@ -0,0 +1,83 @@ +/* + * Copyright 2022 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.google.cloud.datastore.aggregation; + +import com.google.api.core.BetaApi; +import com.google.datastore.v1.AggregationQuery; +import com.google.datastore.v1.AggregationQuery.Aggregation.Count; +import java.util.Objects; + +/** Represents an {@link Aggregation} which returns count. */ +@BetaApi +public class CountAggregation extends Aggregation { + + /** @param alias Alias to used when running this aggregation. */ + public CountAggregation(String alias) { + super(alias); + } + + @Override + public AggregationQuery.Aggregation toPb() { + Count.Builder countBuilder = Count.newBuilder(); + + AggregationQuery.Aggregation.Builder aggregationBuilder = + AggregationQuery.Aggregation.newBuilder().setCount(countBuilder); + if (this.getAlias() != null) { + aggregationBuilder.setAlias(this.getAlias()); + } + return aggregationBuilder.build(); + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (o == null || getClass() != o.getClass()) { + return false; + } + CountAggregation that = (CountAggregation) o; + boolean bothAliasAreNull = getAlias() == null && that.getAlias() == null; + if (bothAliasAreNull) { + return true; + } else { + boolean bothArePresent = getAlias() != null && that.getAlias() != null; + return bothArePresent && getAlias().equals(that.getAlias()); + } + } + + @Override + public int hashCode() { + return Objects.hash(getAlias()); + } + + /** A builder class to create and customize a {@link CountAggregation}. */ + public static class Builder implements AggregationBuilder { + + private String alias; + + public Builder as(String alias) { + this.alias = alias; + return this; + } + + @Override + public CountAggregation build() { + return new CountAggregation(alias); + } + } +} diff --git a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/execution/AggregationQueryExecutor.java b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/execution/AggregationQueryExecutor.java new file mode 100644 index 000000000..14e425845 --- /dev/null +++ b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/execution/AggregationQueryExecutor.java @@ -0,0 +1,66 @@ +/* + * Copyright 2022 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.google.cloud.datastore.execution; + +import com.google.api.core.InternalApi; +import com.google.cloud.datastore.AggregationQuery; +import com.google.cloud.datastore.AggregationResults; +import com.google.cloud.datastore.DatastoreOptions; +import com.google.cloud.datastore.ReadOption; +import com.google.cloud.datastore.ReadOption.QueryAndReadOptions; +import com.google.cloud.datastore.execution.request.AggregationQueryRequestProtoPreparer; +import com.google.cloud.datastore.execution.response.AggregationQueryResponseTransformer; +import com.google.cloud.datastore.spi.v1.DatastoreRpc; +import com.google.datastore.v1.RunAggregationQueryRequest; +import com.google.datastore.v1.RunAggregationQueryResponse; +import java.util.Arrays; + +/** + * An implementation of {@link QueryExecutor} which executes {@link AggregationQuery} and returns + * {@link AggregationResults}. + */ +@InternalApi +public class AggregationQueryExecutor + implements QueryExecutor { + + private final DatastoreRpc datastoreRpc; + private final AggregationQueryRequestProtoPreparer protoPreparer; + private final AggregationQueryResponseTransformer responseTransformer; + + public AggregationQueryExecutor(DatastoreRpc datastoreRpc, DatastoreOptions datastoreOptions) { + this.datastoreRpc = datastoreRpc; + this.protoPreparer = new AggregationQueryRequestProtoPreparer(datastoreOptions); + this.responseTransformer = new AggregationQueryResponseTransformer(); + } + + @Override + public AggregationResults execute(AggregationQuery query, ReadOption... readOptions) { + RunAggregationQueryRequest runAggregationQueryRequest = + getRunAggregationQueryRequest(query, readOptions); + RunAggregationQueryResponse runAggregationQueryResponse = + this.datastoreRpc.runAggregationQuery(runAggregationQueryRequest); + return this.responseTransformer.transform(runAggregationQueryResponse); + } + + private RunAggregationQueryRequest getRunAggregationQueryRequest( + AggregationQuery query, ReadOption... readOptions) { + QueryAndReadOptions queryAndReadOptions = + readOptions == null + ? QueryAndReadOptions.create(query) + : QueryAndReadOptions.create(query, Arrays.asList(readOptions)); + return this.protoPreparer.prepare(queryAndReadOptions); + } +} diff --git a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/execution/QueryExecutor.java b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/execution/QueryExecutor.java new file mode 100644 index 000000000..856c64a02 --- /dev/null +++ b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/execution/QueryExecutor.java @@ -0,0 +1,40 @@ +/* + * Copyright 2022 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.google.cloud.datastore.execution; + +import com.google.api.core.InternalApi; +import com.google.cloud.datastore.Query; +import com.google.cloud.datastore.ReadOption; + +/** + * An internal functional interface whose implementation has the responsibility to execute a {@link + * Query} and returns the result. This class will have the responsibility to orchestrate between + * {@link com.google.cloud.datastore.execution.request.ProtoPreparer}, {@link + * com.google.cloud.datastore.spi.v1.DatastoreRpc} and {@link + * com.google.cloud.datastore.execution.response.ResponseTransformer} layers. + * + * @param A {@link Query} to execute. + * @param the type of result produced by Query. + */ +@InternalApi +public interface QueryExecutor, OUTPUT> { + + /** + * @param query A {@link Query} to execute. + * @param readOptions Optional {@link ReadOption}s to be used when executing {@link Query}. + */ + OUTPUT execute(INPUT query, ReadOption... readOptions); +} diff --git a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/execution/request/AggregationQueryRequestProtoPreparer.java b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/execution/request/AggregationQueryRequestProtoPreparer.java new file mode 100644 index 000000000..b5da8d9fe --- /dev/null +++ b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/execution/request/AggregationQueryRequestProtoPreparer.java @@ -0,0 +1,100 @@ +/* + * Copyright 2022 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.google.cloud.datastore.execution.request; + +import static com.google.cloud.datastore.AggregationQuery.Mode.GQL; + +import com.google.api.core.InternalApi; +import com.google.cloud.datastore.AggregationQuery; +import com.google.cloud.datastore.DatastoreOptions; +import com.google.cloud.datastore.GqlQueryProtoPreparer; +import com.google.cloud.datastore.ReadOption; +import com.google.cloud.datastore.ReadOption.QueryAndReadOptions; +import com.google.cloud.datastore.ReadOptionProtoPreparer; +import com.google.cloud.datastore.StructuredQueryProtoPreparer; +import com.google.cloud.datastore.aggregation.Aggregation; +import com.google.datastore.v1.GqlQuery; +import com.google.datastore.v1.PartitionId; +import com.google.datastore.v1.Query; +import com.google.datastore.v1.ReadOptions; +import com.google.datastore.v1.RunAggregationQueryRequest; +import java.util.List; +import java.util.Optional; + +@InternalApi +public class AggregationQueryRequestProtoPreparer + implements ProtoPreparer, RunAggregationQueryRequest> { + + private final DatastoreOptions datastoreOptions; + private final StructuredQueryProtoPreparer structuredQueryProtoPreparer; + private final GqlQueryProtoPreparer gqlQueryProtoPreparer; + private final ReadOptionProtoPreparer readOptionProtoPreparer; + + public AggregationQueryRequestProtoPreparer(DatastoreOptions datastoreOptions) { + this.datastoreOptions = datastoreOptions; + this.structuredQueryProtoPreparer = new StructuredQueryProtoPreparer(); + this.gqlQueryProtoPreparer = new GqlQueryProtoPreparer(); + this.readOptionProtoPreparer = new ReadOptionProtoPreparer(); + } + + @Override + public RunAggregationQueryRequest prepare( + QueryAndReadOptions aggregationQueryAndReadOptions) { + AggregationQuery aggregationQuery = aggregationQueryAndReadOptions.getQuery(); + List readOptions = aggregationQueryAndReadOptions.getReadOptions(); + PartitionId partitionId = getPartitionId(aggregationQuery); + RunAggregationQueryRequest.Builder aggregationQueryRequestBuilder = + RunAggregationQueryRequest.newBuilder() + .setPartitionId(partitionId) + .setProjectId(datastoreOptions.getProjectId()); + + if (aggregationQuery.getMode() == GQL) { + aggregationQueryRequestBuilder.setGqlQuery(buildGqlQuery(aggregationQuery)); + } else { + aggregationQueryRequestBuilder.setAggregationQuery(getAggregationQuery(aggregationQuery)); + } + + Optional readOptionsPb = readOptionProtoPreparer.prepare(readOptions); + readOptionsPb.ifPresent(aggregationQueryRequestBuilder::setReadOptions); + return aggregationQueryRequestBuilder.build(); + } + + private GqlQuery buildGqlQuery(AggregationQuery aggregationQuery) { + return gqlQueryProtoPreparer.prepare(aggregationQuery.getNestedGqlQuery()); + } + + private com.google.datastore.v1.AggregationQuery getAggregationQuery( + AggregationQuery aggregationQuery) { + Query nestedQueryProto = + structuredQueryProtoPreparer.prepare(aggregationQuery.getNestedStructuredQuery()); + + com.google.datastore.v1.AggregationQuery.Builder aggregationQueryProtoBuilder = + com.google.datastore.v1.AggregationQuery.newBuilder().setNestedQuery(nestedQueryProto); + for (Aggregation aggregation : aggregationQuery.getAggregations()) { + aggregationQueryProtoBuilder.addAggregations(aggregation.toPb()); + } + return aggregationQueryProtoBuilder.build(); + } + + private PartitionId getPartitionId(AggregationQuery aggregationQuery) { + PartitionId.Builder builder = + PartitionId.newBuilder().setProjectId(datastoreOptions.getProjectId()); + if (aggregationQuery.getNamespace() != null) { + builder.setNamespaceId(aggregationQuery.getNamespace()); + } + return builder.build(); + } +} diff --git a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/execution/response/AggregationQueryResponseTransformer.java b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/execution/response/AggregationQueryResponseTransformer.java new file mode 100644 index 000000000..1515a1147 --- /dev/null +++ b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/execution/response/AggregationQueryResponseTransformer.java @@ -0,0 +1,58 @@ +/* + * Copyright 2022 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.google.cloud.datastore.execution.response; + +import com.google.api.core.InternalApi; +import com.google.cloud.Timestamp; +import com.google.cloud.datastore.AggregationResult; +import com.google.cloud.datastore.AggregationResults; +import com.google.cloud.datastore.LongValue; +import com.google.datastore.v1.RunAggregationQueryResponse; +import com.google.datastore.v1.Value; +import java.util.AbstractMap.SimpleEntry; +import java.util.LinkedList; +import java.util.List; +import java.util.Map; +import java.util.Map.Entry; +import java.util.function.Function; +import java.util.stream.Collectors; + +@InternalApi +public class AggregationQueryResponseTransformer + implements ResponseTransformer { + + @Override + public AggregationResults transform(RunAggregationQueryResponse response) { + Timestamp readTime = Timestamp.fromProto(response.getBatch().getReadTime()); + List aggregationResults = + response.getBatch().getAggregationResultsList().stream() + .map( + aggregationResult -> new AggregationResult(resultWithLongValues(aggregationResult))) + .collect(Collectors.toCollection(LinkedList::new)); + return new AggregationResults(aggregationResults, readTime); + } + + private Map resultWithLongValues( + com.google.datastore.v1.AggregationResult aggregationResult) { + return aggregationResult.getAggregatePropertiesMap().entrySet().stream() + .map( + (Function, Entry>) + entry -> + new SimpleEntry<>( + entry.getKey(), (LongValue) LongValue.fromPb(entry.getValue()))) + .collect(Collectors.toMap(Entry::getKey, Entry::getValue)); + } +} diff --git a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/execution/response/ResponseTransformer.java b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/execution/response/ResponseTransformer.java new file mode 100644 index 000000000..b17da3f79 --- /dev/null +++ b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/execution/response/ResponseTransformer.java @@ -0,0 +1,30 @@ +/* + * Copyright 2022 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.google.cloud.datastore.execution.response; + +import com.google.api.core.InternalApi; + +/** + * An internal functional interface whose implementation has the responsibility to populate a Domain + * object from a proto response. + * + * @param the type of proto response object. + * @param the type of domain object. + */ +@InternalApi +public interface ResponseTransformer { + OUTPUT transform(INPUT response); +} diff --git a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/spi/v1/DatastoreRpc.java b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/spi/v1/DatastoreRpc.java index 5e64c9255..33b8e11ea 100644 --- a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/spi/v1/DatastoreRpc.java +++ b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/spi/v1/DatastoreRpc.java @@ -30,6 +30,8 @@ import com.google.datastore.v1.ReserveIdsResponse; import com.google.datastore.v1.RollbackRequest; import com.google.datastore.v1.RollbackResponse; +import com.google.datastore.v1.RunAggregationQueryRequest; +import com.google.datastore.v1.RunAggregationQueryResponse; import com.google.datastore.v1.RunQueryRequest; import com.google.datastore.v1.RunQueryResponse; @@ -85,4 +87,13 @@ BeginTransactionResponse beginTransaction(BeginTransactionRequest request) * @throws DatastoreException upon failure */ RunQueryResponse runQuery(RunQueryRequest request); + + /** + * Sends a request to run an aggregation query. + * + * @throws DatastoreException upon failure + */ + default RunAggregationQueryResponse runAggregationQuery(RunAggregationQueryRequest request) { + throw new UnsupportedOperationException("Not implemented."); + } } diff --git a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/spi/v1/HttpDatastoreRpc.java b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/spi/v1/HttpDatastoreRpc.java index 4f13b4600..fd3cdc658 100644 --- a/google-cloud-datastore/src/main/java/com/google/cloud/datastore/spi/v1/HttpDatastoreRpc.java +++ b/google-cloud-datastore/src/main/java/com/google/cloud/datastore/spi/v1/HttpDatastoreRpc.java @@ -36,6 +36,8 @@ import com.google.datastore.v1.ReserveIdsResponse; import com.google.datastore.v1.RollbackRequest; import com.google.datastore.v1.RollbackResponse; +import com.google.datastore.v1.RunAggregationQueryRequest; +import com.google.datastore.v1.RunAggregationQueryResponse; import com.google.datastore.v1.RunQueryRequest; import com.google.datastore.v1.RunQueryResponse; import java.io.IOException; @@ -200,4 +202,13 @@ public RunQueryResponse runQuery(RunQueryRequest request) { throw translate(ex); } } + + @Override + public RunAggregationQueryResponse runAggregationQuery(RunAggregationQueryRequest request) { + try { + return client.runAggregationQuery(request); + } catch (com.google.datastore.v1.client.DatastoreException ex) { + throw translate(ex); + } + } } diff --git a/google-cloud-datastore/src/test/java/com/google/cloud/datastore/AggregationQueryTest.java b/google-cloud-datastore/src/test/java/com/google/cloud/datastore/AggregationQueryTest.java new file mode 100644 index 000000000..840d23bca --- /dev/null +++ b/google-cloud-datastore/src/test/java/com/google/cloud/datastore/AggregationQueryTest.java @@ -0,0 +1,153 @@ +/* + * Copyright 2022 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.google.cloud.datastore; + +import static com.google.cloud.datastore.AggregationQuery.Mode.GQL; +import static com.google.cloud.datastore.AggregationQuery.Mode.STRUCTURED; +import static com.google.cloud.datastore.StructuredQuery.PropertyFilter.eq; +import static com.google.cloud.datastore.aggregation.Aggregation.count; +import static com.google.common.truth.Truth.assertThat; +import static org.junit.Assert.assertNull; +import static org.junit.Assert.assertThrows; + +import com.google.cloud.datastore.aggregation.CountAggregation; +import com.google.common.collect.ImmutableSet; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.ExpectedException; + +public class AggregationQueryTest { + + private static final String KIND = "Task"; + private static final String NAMESPACE = "ns"; + private static final EntityQuery COMPLETED_TASK_QUERY = + Query.newEntityQueryBuilder() + .setNamespace(NAMESPACE) + .setKind(KIND) + .setFilter(eq("done", true)) + .setLimit(100) + .build(); + + @Rule public ExpectedException exceptionRule = ExpectedException.none(); + + @Test + public void testAggregations() { + AggregationQuery aggregationQuery = + Query.newAggregationQueryBuilder() + .setNamespace(NAMESPACE) + .addAggregation(new CountAggregation("total")) + .over(COMPLETED_TASK_QUERY) + .build(); + + assertThat(aggregationQuery.getNamespace()).isEqualTo(NAMESPACE); + assertThat(aggregationQuery.getAggregations()) + .isEqualTo(ImmutableSet.of(count().as("total").build())); + assertThat(aggregationQuery.getNestedStructuredQuery()).isEqualTo(COMPLETED_TASK_QUERY); + assertThat(aggregationQuery.getMode()).isEqualTo(STRUCTURED); + } + + @Test + public void testAggregationBuilderWithMoreThanOneAggregations() { + AggregationQuery aggregationQuery = + Query.newAggregationQueryBuilder() + .setNamespace(NAMESPACE) + .addAggregation(count().as("total")) + .addAggregation(count().as("new_total")) + .over(COMPLETED_TASK_QUERY) + .build(); + + assertThat(aggregationQuery.getNamespace()).isEqualTo(NAMESPACE); + assertThat(aggregationQuery.getAggregations()) + .isEqualTo(ImmutableSet.of(count().as("total").build(), count().as("new_total").build())); + assertThat(aggregationQuery.getNestedStructuredQuery()).isEqualTo(COMPLETED_TASK_QUERY); + assertThat(aggregationQuery.getMode()).isEqualTo(STRUCTURED); + } + + @Test + public void testAggregationBuilderWithDuplicateAggregations() { + AggregationQuery aggregationQuery = + Query.newAggregationQueryBuilder() + .setNamespace(NAMESPACE) + .addAggregation(count().as("total")) + .addAggregation(count().as("total")) + .over(COMPLETED_TASK_QUERY) + .build(); + + assertThat(aggregationQuery.getNamespace()).isEqualTo(NAMESPACE); + assertThat(aggregationQuery.getAggregations()) + .isEqualTo(ImmutableSet.of(count().as("total").build())); + assertThat(aggregationQuery.getNestedStructuredQuery()).isEqualTo(COMPLETED_TASK_QUERY); + assertThat(aggregationQuery.getMode()).isEqualTo(STRUCTURED); + } + + @Test + public void testAggregationQueryBuilderWithoutNamespace() { + AggregationQuery aggregationQuery = + Query.newAggregationQueryBuilder() + .addAggregation(count().as("total")) + .over(COMPLETED_TASK_QUERY) + .build(); + + assertNull(aggregationQuery.getNamespace()); + assertThat(aggregationQuery.getAggregations()) + .isEqualTo(ImmutableSet.of(count().as("total").build())); + assertThat(aggregationQuery.getNestedStructuredQuery()).isEqualTo(COMPLETED_TASK_QUERY); + assertThat(aggregationQuery.getMode()).isEqualTo(STRUCTURED); + } + + @Test + public void testAggregationQueryBuilderWithoutNestedQuery() { + assertThrows( + "Nested query is required for an aggregation query to run", + IllegalArgumentException.class, + () -> + Query.newAggregationQueryBuilder() + .setNamespace(NAMESPACE) + .addAggregation(count().as("total")) + .build()); + } + + @Test + public void testAggregationQueryBuilderWithoutAggregation() { + assertThrows( + "At least one aggregation is required for an aggregation query to run", + IllegalArgumentException.class, + () -> + Query.newAggregationQueryBuilder() + .setNamespace(NAMESPACE) + .over(COMPLETED_TASK_QUERY) + .build()); + } + + @Test + public void testAggregationQueryBuilderWithGqlQuery() { + GqlQuery gqlQuery = Query.newGqlQueryBuilder("SELECT * FROM Task WHERE done = true").build(); + + AggregationQuery aggregationQuery = + Query.newAggregationQueryBuilder().setNamespace(NAMESPACE).over(gqlQuery).build(); + + assertThat(aggregationQuery.getNestedGqlQuery()).isEqualTo(gqlQuery); + assertThat(aggregationQuery.getMode()).isEqualTo(GQL); + } + + @Test + public void testAggregationQueryBuilderWithoutProvidingAnyNestedQuery() { + assertThrows( + "Nested query is required for an aggregation query to run", + IllegalArgumentException.class, + () -> Query.newAggregationQueryBuilder().setNamespace(NAMESPACE).build()); + } +} diff --git a/google-cloud-datastore/src/test/java/com/google/cloud/datastore/AggregationResultTest.java b/google-cloud-datastore/src/test/java/com/google/cloud/datastore/AggregationResultTest.java new file mode 100644 index 000000000..06a5cb5f7 --- /dev/null +++ b/google-cloud-datastore/src/test/java/com/google/cloud/datastore/AggregationResultTest.java @@ -0,0 +1,36 @@ +/* + * Copyright 2022 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.google.cloud.datastore; + +import static com.google.common.truth.Truth.assertThat; + +import com.google.common.collect.ImmutableMap; +import org.junit.Test; + +public class AggregationResultTest { + + @Test + public void shouldGetAggregationResultValueByAlias() { + AggregationResult aggregationResult = + new AggregationResult( + ImmutableMap.of( + "count", LongValue.of(45), + "property_2", LongValue.of(30))); + + assertThat(aggregationResult.get("count")).isEqualTo(45L); + assertThat(aggregationResult.get("property_2")).isEqualTo(30L); + } +} diff --git a/google-cloud-datastore/src/test/java/com/google/cloud/datastore/DatastoreTest.java b/google-cloud-datastore/src/test/java/com/google/cloud/datastore/DatastoreTest.java index 72067fd20..7dc625bad 100644 --- a/google-cloud-datastore/src/test/java/com/google/cloud/datastore/DatastoreTest.java +++ b/google-cloud-datastore/src/test/java/com/google/cloud/datastore/DatastoreTest.java @@ -16,6 +16,11 @@ package com.google.cloud.datastore; +import static com.google.cloud.datastore.ProtoTestData.intValue; +import static com.google.cloud.datastore.TestUtils.matches; +import static com.google.cloud.datastore.aggregation.Aggregation.count; +import static com.google.common.collect.Iterables.getOnlyElement; +import static com.google.common.truth.Truth.assertThat; import static org.easymock.EasyMock.createStrictMock; import static org.easymock.EasyMock.replay; import static org.easymock.EasyMock.verify; @@ -37,8 +42,10 @@ import com.google.cloud.datastore.spi.v1.DatastoreRpc; import com.google.cloud.datastore.testing.LocalDatastoreHelper; import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableMap; import com.google.common.collect.Iterators; import com.google.common.collect.Lists; +import com.google.datastore.v1.AggregationResultBatch; import com.google.datastore.v1.BeginTransactionRequest; import com.google.datastore.v1.BeginTransactionResponse; import com.google.datastore.v1.CommitRequest; @@ -54,6 +61,8 @@ import com.google.datastore.v1.ReserveIdsResponse; import com.google.datastore.v1.RollbackRequest; import com.google.datastore.v1.RollbackResponse; +import com.google.datastore.v1.RunAggregationQueryRequest; +import com.google.datastore.v1.RunAggregationQueryResponse; import com.google.datastore.v1.RunQueryRequest; import com.google.datastore.v1.RunQueryResponse; import com.google.datastore.v1.TransactionOptions; @@ -62,11 +71,14 @@ import java.util.ArrayList; import java.util.Arrays; import java.util.Collections; +import java.util.HashMap; import java.util.HashSet; import java.util.Iterator; import java.util.List; +import java.util.Map; import java.util.Set; import java.util.concurrent.TimeoutException; +import java.util.function.Predicate; import org.easymock.EasyMock; import org.junit.AfterClass; import org.junit.Assert; @@ -526,6 +538,27 @@ public void testGqlQueryPagination() throws DatastoreException { EasyMock.verify(rpcFactoryMock, rpcMock); } + @Test + public void testRunAggregationQuery() { + RunAggregationQueryResponse aggregationQueryResponse = placeholderAggregationQueryResponse(); + EasyMock.expect(rpcMock.runAggregationQuery(matches(aggregationQueryWithAlias("total_count")))) + .andReturn(aggregationQueryResponse); + EasyMock.replay(rpcFactoryMock, rpcMock); + + Datastore mockDatastore = rpcMockOptions.getService(); + + EntityQuery selectAllQuery = Query.newEntityQueryBuilder().build(); + AggregationQuery getCountQuery = + Query.newAggregationQueryBuilder() + .addAggregation(count().as("total_count")) + .over(selectAllQuery) + .build(); + AggregationResult result = getOnlyElement(mockDatastore.runAggregation(getCountQuery)); + + assertThat(result.get("total_count")).isEqualTo(209L); + EasyMock.verify(rpcFactoryMock, rpcMock); + } + @Test public void testRunStructuredQuery() { Query query = @@ -1311,4 +1344,28 @@ public void testQueryWithStartCursor() { assertEquals(cursor2, cursor1); datastore.delete(entity1.getKey(), entity2.getKey(), entity3.getKey()); } + + private RunAggregationQueryResponse placeholderAggregationQueryResponse() { + Map result1 = + new HashMap<>(ImmutableMap.of("total_count", intValue(209))); + + AggregationResultBatch resultBatch = + AggregationResultBatch.newBuilder() + .addAggregationResults( + com.google.datastore.v1.AggregationResult.newBuilder() + .putAllAggregateProperties(result1) + .build()) + .build(); + return RunAggregationQueryResponse.newBuilder().setBatch(resultBatch).build(); + } + + private Predicate aggregationQueryWithAlias(String alias) { + return runAggregationQueryRequest -> + alias.equals( + runAggregationQueryRequest + .getAggregationQuery() + .getAggregationsList() + .get(0) + .getAlias()); + } } diff --git a/google-cloud-datastore/src/test/java/com/google/cloud/datastore/ProtoTestData.java b/google-cloud-datastore/src/test/java/com/google/cloud/datastore/ProtoTestData.java index a923b618b..25b902fd4 100644 --- a/google-cloud-datastore/src/test/java/com/google/cloud/datastore/ProtoTestData.java +++ b/google-cloud-datastore/src/test/java/com/google/cloud/datastore/ProtoTestData.java @@ -17,6 +17,8 @@ import static com.google.datastore.v1.PropertyOrder.Direction.ASCENDING; +import com.google.datastore.v1.AggregationQuery.Aggregation; +import com.google.datastore.v1.AggregationQuery.Aggregation.Count; import com.google.datastore.v1.Filter; import com.google.datastore.v1.GqlQueryParameter; import com.google.datastore.v1.KindExpression; @@ -63,6 +65,10 @@ public static PropertyReference propertyReference(String value) { return PropertyReference.newBuilder().setName(value).build(); } + public static Aggregation countAggregation(String alias) { + return Aggregation.newBuilder().setAlias(alias).setCount(Count.newBuilder().build()).build(); + } + public static PropertyOrder propertyOrder(String value) { return PropertyOrder.newBuilder() .setProperty(propertyReference(value)) diff --git a/google-cloud-datastore/src/test/java/com/google/cloud/datastore/ReadOptionProtoPreparerTest.java b/google-cloud-datastore/src/test/java/com/google/cloud/datastore/ReadOptionProtoPreparerTest.java index bda5de3b5..b16fdf100 100644 --- a/google-cloud-datastore/src/test/java/com/google/cloud/datastore/ReadOptionProtoPreparerTest.java +++ b/google-cloud-datastore/src/test/java/com/google/cloud/datastore/ReadOptionProtoPreparerTest.java @@ -78,11 +78,11 @@ public void shouldPrepareReadOptionsWithReadTime() { @Test public void shouldPrepareReadOptionsWithTransactionId() { - String dummyTransactionId = "transaction-id"; + String transactionId = "transaction-id"; Optional readOptions = - protoPreparer.prepare(singletonList(transactionId(dummyTransactionId))); + protoPreparer.prepare(singletonList(transactionId(transactionId))); - assertThat(readOptions.get().getTransaction().toStringUtf8()).isEqualTo(dummyTransactionId); + assertThat(readOptions.get().getTransaction().toStringUtf8()).isEqualTo(transactionId); } @Test diff --git a/google-cloud-datastore/src/test/java/com/google/cloud/datastore/RetryAndTraceDatastoreRpcDecoratorTest.java b/google-cloud-datastore/src/test/java/com/google/cloud/datastore/RetryAndTraceDatastoreRpcDecoratorTest.java new file mode 100644 index 000000000..b86355afa --- /dev/null +++ b/google-cloud-datastore/src/test/java/com/google/cloud/datastore/RetryAndTraceDatastoreRpcDecoratorTest.java @@ -0,0 +1,84 @@ +/* + * Copyright 2022 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.google.cloud.datastore; + +import static com.google.cloud.datastore.TraceUtil.END_SPAN_OPTIONS; +import static com.google.cloud.datastore.TraceUtil.SPAN_NAME_RUN_AGGREGATION_QUERY; +import static com.google.common.truth.Truth.assertThat; +import static com.google.rpc.Code.UNAVAILABLE; +import static org.easymock.EasyMock.createNiceMock; +import static org.easymock.EasyMock.createStrictMock; +import static org.easymock.EasyMock.expect; +import static org.easymock.EasyMock.replay; +import static org.easymock.EasyMock.verify; + +import com.google.api.gax.retrying.RetrySettings; +import com.google.cloud.datastore.spi.v1.DatastoreRpc; +import com.google.datastore.v1.RunAggregationQueryRequest; +import com.google.datastore.v1.RunAggregationQueryResponse; +import io.opencensus.trace.Span; +import io.opencensus.trace.Tracer; +import org.junit.Before; +import org.junit.Test; + +public class RetryAndTraceDatastoreRpcDecoratorTest { + + public static final int MAX_ATTEMPTS = 3; + private DatastoreRpc mockDatastoreRpc; + private TraceUtil mockTraceUtil; + private DatastoreOptions datastoreOptions = + DatastoreOptions.newBuilder().setProjectId("project-id").build(); + private RetrySettings retrySettings = + RetrySettings.newBuilder().setMaxAttempts(MAX_ATTEMPTS).build(); + + private RetryAndTraceDatastoreRpcDecorator datastoreRpcDecorator; + + @Before + public void setUp() throws Exception { + mockDatastoreRpc = createStrictMock(DatastoreRpc.class); + mockTraceUtil = createStrictMock(TraceUtil.class); + datastoreRpcDecorator = + new RetryAndTraceDatastoreRpcDecorator( + mockDatastoreRpc, mockTraceUtil, retrySettings, datastoreOptions); + } + + @Test + public void testRunAggregationQuery() { + Span mockSpan = createStrictMock(Span.class); + RunAggregationQueryRequest aggregationQueryRequest = + RunAggregationQueryRequest.getDefaultInstance(); + RunAggregationQueryResponse aggregationQueryResponse = + RunAggregationQueryResponse.getDefaultInstance(); + + expect(mockDatastoreRpc.runAggregationQuery(aggregationQueryRequest)) + .andThrow( + new DatastoreException( + UNAVAILABLE.getNumber(), "API not accessible currently", UNAVAILABLE.name())) + .times(2) + .andReturn(aggregationQueryResponse); + expect(mockTraceUtil.startSpan(SPAN_NAME_RUN_AGGREGATION_QUERY)).andReturn(mockSpan); + expect(mockTraceUtil.getTracer()).andReturn(createNiceMock(Tracer.class)); + mockSpan.end(END_SPAN_OPTIONS); + + replay(mockDatastoreRpc, mockTraceUtil, mockSpan); + + RunAggregationQueryResponse actualAggregationQueryResponse = + datastoreRpcDecorator.runAggregationQuery(aggregationQueryRequest); + + assertThat(actualAggregationQueryResponse).isSameInstanceAs(aggregationQueryResponse); + verify(mockDatastoreRpc, mockTraceUtil, mockSpan); + } +} diff --git a/google-cloud-datastore/src/test/java/com/google/cloud/datastore/TestUtils.java b/google-cloud-datastore/src/test/java/com/google/cloud/datastore/TestUtils.java new file mode 100644 index 000000000..3a3fcfaea --- /dev/null +++ b/google-cloud-datastore/src/test/java/com/google/cloud/datastore/TestUtils.java @@ -0,0 +1,39 @@ +/* + * Copyright 2022 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.google.cloud.datastore; + +import java.util.function.Predicate; +import org.easymock.EasyMock; +import org.easymock.IArgumentMatcher; + +public class TestUtils { + + public static T matches(Predicate predicate) { + EasyMock.reportMatcher( + new IArgumentMatcher() { + @Override + public boolean matches(Object argument) { + return predicate.test(((T) argument)); + } + + @Override + public void appendTo(StringBuffer buffer) { + buffer.append("matches(\"").append(predicate).append("\")"); + } + }); + return null; + } +} diff --git a/google-cloud-datastore/src/test/java/com/google/cloud/datastore/aggregation/CountAggregationTest.java b/google-cloud-datastore/src/test/java/com/google/cloud/datastore/aggregation/CountAggregationTest.java new file mode 100644 index 000000000..a8b3bc945 --- /dev/null +++ b/google-cloud-datastore/src/test/java/com/google/cloud/datastore/aggregation/CountAggregationTest.java @@ -0,0 +1,68 @@ +/* + * Copyright 2022 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.google.cloud.datastore.aggregation; + +import static com.google.cloud.datastore.aggregation.Aggregation.count; +import static com.google.common.truth.Truth.assertThat; + +import com.google.datastore.v1.AggregationQuery; +import org.junit.Test; + +public class CountAggregationTest { + + @Test + public void testCountAggregationWithDefaultValues() { + AggregationQuery.Aggregation countAggregationPb = count().build().toPb(); + + assertThat(countAggregationPb.getCount().getUpTo().getValue()).isEqualTo(0L); + assertThat(countAggregationPb.getAlias()).isEqualTo(""); + } + + @Test + public void testCountAggregationWithAlias() { + AggregationQuery.Aggregation countAggregationPb = count().as("column_1").build().toPb(); + + assertThat(countAggregationPb.getCount().getUpTo().getValue()).isEqualTo(0L); + assertThat(countAggregationPb.getAlias()).isEqualTo("column_1"); + } + + @Test + public void testEquals() { + CountAggregation.Builder aggregationWithAlias1 = count().as("total"); + CountAggregation.Builder aggregationWithAlias2 = count().as("total"); + CountAggregation.Builder aggregationWithoutAlias1 = count(); + CountAggregation.Builder aggregationWithoutAlias2 = count(); + + // same aliases + assertThat(aggregationWithAlias1.build()).isEqualTo(aggregationWithAlias2.build()); + assertThat(aggregationWithAlias2.build()).isEqualTo(aggregationWithAlias1.build()); + + // with and without aliases + assertThat(aggregationWithAlias1.build()).isNotEqualTo(aggregationWithoutAlias1.build()); + assertThat(aggregationWithoutAlias1.build()).isNotEqualTo(aggregationWithAlias1.build()); + + // no aliases + assertThat(aggregationWithoutAlias1.build()).isEqualTo(aggregationWithoutAlias2.build()); + assertThat(aggregationWithoutAlias2.build()).isEqualTo(aggregationWithoutAlias1.build()); + + // different aliases + assertThat(aggregationWithAlias1.as("new-alias").build()) + .isNotEqualTo(aggregationWithAlias2.build()); + assertThat(aggregationWithAlias2.build()) + .isNotEqualTo(aggregationWithAlias1.as("new-alias").build()); + } +} diff --git a/google-cloud-datastore/src/test/java/com/google/cloud/datastore/execution/AggregationQueryExecutorTest.java b/google-cloud-datastore/src/test/java/com/google/cloud/datastore/execution/AggregationQueryExecutorTest.java new file mode 100644 index 000000000..f9f23261d --- /dev/null +++ b/google-cloud-datastore/src/test/java/com/google/cloud/datastore/execution/AggregationQueryExecutorTest.java @@ -0,0 +1,177 @@ +/* + * Copyright 2022 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.google.cloud.datastore.execution; + +import static com.google.cloud.datastore.ProtoTestData.intValue; +import static com.google.cloud.datastore.ReadOption.eventualConsistency; +import static com.google.cloud.datastore.StructuredQuery.PropertyFilter.eq; +import static com.google.cloud.datastore.TestUtils.matches; +import static com.google.cloud.datastore.aggregation.Aggregation.count; +import static com.google.common.truth.Truth.assertThat; +import static com.google.datastore.v1.ReadOptions.ReadConsistency.EVENTUAL; +import static java.util.Arrays.asList; +import static org.easymock.EasyMock.anyObject; +import static org.easymock.EasyMock.expect; +import static org.easymock.EasyMock.replay; +import static org.easymock.EasyMock.verify; + +import com.google.cloud.Timestamp; +import com.google.cloud.datastore.AggregationQuery; +import com.google.cloud.datastore.AggregationResult; +import com.google.cloud.datastore.AggregationResults; +import com.google.cloud.datastore.DatastoreOptions; +import com.google.cloud.datastore.EntityQuery; +import com.google.cloud.datastore.LongValue; +import com.google.cloud.datastore.Query; +import com.google.cloud.datastore.spi.v1.DatastoreRpc; +import com.google.common.collect.ImmutableMap; +import com.google.datastore.v1.AggregationResultBatch; +import com.google.datastore.v1.RunAggregationQueryRequest; +import com.google.datastore.v1.RunAggregationQueryResponse; +import com.google.datastore.v1.Value; +import java.util.HashMap; +import java.util.Map; +import java.util.function.Predicate; +import org.easymock.EasyMock; +import org.junit.Before; +import org.junit.Test; + +public class AggregationQueryExecutorTest { + + private static final String KIND = "Task"; + private static final String NAMESPACE = "ns"; + + private DatastoreRpc mockRpc; + private AggregationQueryExecutor queryExecutor; + private DatastoreOptions datastoreOptions; + + @Before + public void setUp() throws Exception { + mockRpc = EasyMock.createStrictMock(DatastoreRpc.class); + datastoreOptions = + DatastoreOptions.newBuilder().setProjectId("project-id").setNamespace(NAMESPACE).build(); + queryExecutor = new AggregationQueryExecutor(mockRpc, datastoreOptions); + } + + @Test + public void shouldExecuteAggregationQuery() { + EntityQuery nestedQuery = + Query.newEntityQueryBuilder() + .setNamespace(NAMESPACE) + .setKind(KIND) + .setFilter(eq("done", true)) + .build(); + + AggregationQuery aggregationQuery = + Query.newAggregationQueryBuilder() + .setNamespace(NAMESPACE) + .addAggregation(count().as("total")) + .over(nestedQuery) + .build(); + + RunAggregationQueryResponse runAggregationQueryResponse = placeholderAggregationQueryResponse(); + expect(mockRpc.runAggregationQuery(anyObject(RunAggregationQueryRequest.class))) + .andReturn(runAggregationQueryResponse); + + replay(mockRpc); + + AggregationResults aggregationResults = queryExecutor.execute(aggregationQuery); + + verify(mockRpc); + assertThat(aggregationResults) + .isEqualTo( + new AggregationResults( + asList( + new AggregationResult( + ImmutableMap.of( + "count", LongValue.of(209), "property_2", LongValue.of(100))), + new AggregationResult( + ImmutableMap.of( + "count", LongValue.of(509), "property_2", LongValue.of(100)))), + Timestamp.fromProto(runAggregationQueryResponse.getBatch().getReadTime()))); + } + + @Test + public void shouldExecuteAggregationQueryWithReadOptions() { + EntityQuery nestedQuery = + Query.newEntityQueryBuilder() + .setNamespace(NAMESPACE) + .setKind(KIND) + .setFilter(eq("done", true)) + .build(); + + AggregationQuery aggregationQuery = + Query.newAggregationQueryBuilder() + .setNamespace(NAMESPACE) + .addAggregation(count().as("total")) + .over(nestedQuery) + .build(); + + RunAggregationQueryResponse runAggregationQueryResponse = placeholderAggregationQueryResponse(); + expect(mockRpc.runAggregationQuery(matches(runAggregationRequestWithEventualConsistency()))) + .andReturn(runAggregationQueryResponse); + + replay(mockRpc); + + AggregationResults aggregationResults = + queryExecutor.execute(aggregationQuery, eventualConsistency()); + + verify(mockRpc); + assertThat(aggregationResults) + .isEqualTo( + new AggregationResults( + asList( + new AggregationResult( + ImmutableMap.of( + "count", LongValue.of(209), "property_2", LongValue.of(100))), + new AggregationResult( + ImmutableMap.of( + "count", LongValue.of(509), "property_2", LongValue.of(100)))), + Timestamp.fromProto(runAggregationQueryResponse.getBatch().getReadTime()))); + } + + private RunAggregationQueryResponse placeholderAggregationQueryResponse() { + Map result1 = + new HashMap<>( + ImmutableMap.of( + "count", intValue(209), + "property_2", intValue(100))); + + Map result2 = + new HashMap<>( + ImmutableMap.of( + "count", intValue(509), + "property_2", intValue(100))); + + AggregationResultBatch resultBatch = + AggregationResultBatch.newBuilder() + .addAggregationResults( + com.google.datastore.v1.AggregationResult.newBuilder() + .putAllAggregateProperties(result1) + .build()) + .addAggregationResults( + com.google.datastore.v1.AggregationResult.newBuilder() + .putAllAggregateProperties(result2) + .build()) + .build(); + return RunAggregationQueryResponse.newBuilder().setBatch(resultBatch).build(); + } + + private Predicate runAggregationRequestWithEventualConsistency() { + return runAggregationQueryRequest -> + runAggregationQueryRequest.getReadOptions().getReadConsistency() == EVENTUAL; + } +} diff --git a/google-cloud-datastore/src/test/java/com/google/cloud/datastore/execution/request/AggregationQueryRequestProtoPreparerTest.java b/google-cloud-datastore/src/test/java/com/google/cloud/datastore/execution/request/AggregationQueryRequestProtoPreparerTest.java new file mode 100644 index 000000000..6301ebeff --- /dev/null +++ b/google-cloud-datastore/src/test/java/com/google/cloud/datastore/execution/request/AggregationQueryRequestProtoPreparerTest.java @@ -0,0 +1,179 @@ +/* + * Copyright 2022 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.google.cloud.datastore.execution.request; + +import static com.google.cloud.datastore.ProtoTestData.booleanValue; +import static com.google.cloud.datastore.ProtoTestData.countAggregation; +import static com.google.cloud.datastore.ProtoTestData.gqlQueryParameter; +import static com.google.cloud.datastore.ProtoTestData.intValue; +import static com.google.cloud.datastore.ProtoTestData.kind; +import static com.google.cloud.datastore.ProtoTestData.propertyFilter; +import static com.google.cloud.datastore.ProtoTestData.stringValue; +import static com.google.cloud.datastore.ReadOption.eventualConsistency; +import static com.google.cloud.datastore.StructuredQuery.PropertyFilter.eq; +import static com.google.cloud.datastore.aggregation.Aggregation.count; +import static com.google.common.truth.Truth.assertThat; +import static com.google.datastore.v1.PropertyFilter.Operator.EQUAL; +import static com.google.datastore.v1.ReadOptions.ReadConsistency.EVENTUAL; +import static java.util.Arrays.asList; +import static java.util.Collections.singletonList; + +import com.google.cloud.Timestamp; +import com.google.cloud.datastore.AggregationQuery; +import com.google.cloud.datastore.DatastoreOptions; +import com.google.cloud.datastore.EntityQuery; +import com.google.cloud.datastore.GqlQuery; +import com.google.cloud.datastore.Query; +import com.google.cloud.datastore.ReadOption; +import com.google.cloud.datastore.ReadOption.QueryAndReadOptions; +import com.google.common.collect.ImmutableMap; +import com.google.datastore.v1.RunAggregationQueryRequest; +import java.util.HashMap; +import org.junit.Test; + +public class AggregationQueryRequestProtoPreparerTest { + + private static final String KIND = "Task"; + private static final String NAMESPACE = "ns"; + private static final String PROJECT_ID = "project-id"; + private static final DatastoreOptions DATASTORE_OPTIONS = + DatastoreOptions.newBuilder().setProjectId(PROJECT_ID).setNamespace(NAMESPACE).build(); + private static final EntityQuery COMPLETED_TASK_STRUCTURED_QUERY = + Query.newEntityQueryBuilder() + .setNamespace(NAMESPACE) + .setKind(KIND) + .setFilter(eq("done", true)) + .build(); + + private static final GqlQuery COMPLETED_TASK_GQL_QUERY = + Query.newGqlQueryBuilder( + "AGGREGATE COUNT AS total_characters OVER (" + + "SELECT * FROM Character WHERE name = @name and age > @1" + + ")") + .setBinding("name", "John Doe") + .addBinding(27) + .build(); + + private final AggregationQuery AGGREGATION_OVER_STRUCTURED_QUERY = + Query.newAggregationQueryBuilder() + .setNamespace(NAMESPACE) + .addAggregation(count().as("total")) + .over(COMPLETED_TASK_STRUCTURED_QUERY) + .build(); + + private final AggregationQuery AGGREGATION_OVER_GQL_QUERY = + Query.newAggregationQueryBuilder() + .setNamespace(NAMESPACE) + .over(COMPLETED_TASK_GQL_QUERY) + .build(); + + private final AggregationQueryRequestProtoPreparer protoPreparer = + new AggregationQueryRequestProtoPreparer(DATASTORE_OPTIONS); + + @Test + public void shouldPrepareAggregationQueryRequestWithGivenStructuredQuery() { + RunAggregationQueryRequest runAggregationQueryRequest = + protoPreparer.prepare(QueryAndReadOptions.create(AGGREGATION_OVER_STRUCTURED_QUERY)); + + assertThat(runAggregationQueryRequest.getProjectId()).isEqualTo(PROJECT_ID); + + assertThat(runAggregationQueryRequest.getPartitionId().getProjectId()).isEqualTo(PROJECT_ID); + assertThat(runAggregationQueryRequest.getPartitionId().getNamespaceId()).isEqualTo(NAMESPACE); + + com.google.datastore.v1.AggregationQuery aggregationQueryProto = + runAggregationQueryRequest.getAggregationQuery(); + assertThat(aggregationQueryProto.getNestedQuery()) + .isEqualTo( + com.google.datastore.v1.Query.newBuilder() + .addKind(kind(KIND)) + .setFilter(propertyFilter("done", EQUAL, booleanValue(true))) + .build()); + assertThat(aggregationQueryProto.getAggregationsList()) + .isEqualTo(singletonList(countAggregation("total"))); + } + + @Test + public void shouldPrepareAggregationQueryRequestWithGivenGqlQuery() { + RunAggregationQueryRequest runAggregationQueryRequest = + protoPreparer.prepare(QueryAndReadOptions.create(AGGREGATION_OVER_GQL_QUERY)); + + assertThat(runAggregationQueryRequest.getProjectId()).isEqualTo(PROJECT_ID); + + assertThat(runAggregationQueryRequest.getPartitionId().getProjectId()).isEqualTo(PROJECT_ID); + assertThat(runAggregationQueryRequest.getPartitionId().getNamespaceId()).isEqualTo(NAMESPACE); + + com.google.datastore.v1.GqlQuery gqlQueryProto = runAggregationQueryRequest.getGqlQuery(); + + assertThat(gqlQueryProto.getQueryString()).isEqualTo(COMPLETED_TASK_GQL_QUERY.getQueryString()); + assertThat(gqlQueryProto.getNamedBindingsMap()) + .isEqualTo( + new HashMap<>(ImmutableMap.of("name", gqlQueryParameter(stringValue("John Doe"))))); + assertThat(gqlQueryProto.getPositionalBindingsList()) + .isEqualTo(asList(gqlQueryParameter(intValue(27)))); + } + + @Test + public void shouldPrepareReadOptionsWithGivenStructuredQuery() { + RunAggregationQueryRequest eventualConsistencyAggregationRequest = + prepareQuery(AGGREGATION_OVER_STRUCTURED_QUERY, eventualConsistency()); + assertThat(eventualConsistencyAggregationRequest.getReadOptions().getReadConsistency()) + .isEqualTo(EVENTUAL); + + Timestamp now = Timestamp.now(); + RunAggregationQueryRequest readTimeAggregationRequest = + prepareQuery(AGGREGATION_OVER_STRUCTURED_QUERY, ReadOption.readTime(now)); + assertThat(Timestamp.fromProto(readTimeAggregationRequest.getReadOptions().getReadTime())) + .isEqualTo(now); + } + + @Test + public void shouldPrepareReadOptionsWithGivenGqlQuery() { + RunAggregationQueryRequest eventualConsistencyAggregationRequest = + prepareQuery(AGGREGATION_OVER_GQL_QUERY, eventualConsistency()); + assertThat(eventualConsistencyAggregationRequest.getReadOptions().getReadConsistency()) + .isEqualTo(EVENTUAL); + + Timestamp now = Timestamp.now(); + RunAggregationQueryRequest readTimeAggregationRequest = + prepareQuery(AGGREGATION_OVER_GQL_QUERY, ReadOption.readTime(now)); + assertThat(Timestamp.fromProto(readTimeAggregationRequest.getReadOptions().getReadTime())) + .isEqualTo(now); + } + + @Test + public void shouldPrepareAggregationQueryWithoutNamespace() { + AggregationQuery structuredQueryWithoutNamespace = + Query.newAggregationQueryBuilder() + .addAggregation(count().as("total")) + .over(COMPLETED_TASK_STRUCTURED_QUERY) + .build(); + AggregationQuery gqlQueryWithoutNamespace = + Query.newAggregationQueryBuilder().over(COMPLETED_TASK_GQL_QUERY).build(); + + RunAggregationQueryRequest runAggregationQueryFromStructuredQuery = + protoPreparer.prepare(QueryAndReadOptions.create(structuredQueryWithoutNamespace)); + RunAggregationQueryRequest runAggregationQueryFromGqlQuery = + protoPreparer.prepare(QueryAndReadOptions.create(gqlQueryWithoutNamespace)); + + assertThat(runAggregationQueryFromStructuredQuery.getPartitionId().getNamespaceId()) + .isEqualTo(""); + assertThat(runAggregationQueryFromGqlQuery.getPartitionId().getNamespaceId()).isEqualTo(""); + } + + private RunAggregationQueryRequest prepareQuery(AggregationQuery query, ReadOption readOption) { + return protoPreparer.prepare(QueryAndReadOptions.create(query, singletonList(readOption))); + } +} diff --git a/google-cloud-datastore/src/test/java/com/google/cloud/datastore/execution/response/AggregationQueryResponseTransformerTest.java b/google-cloud-datastore/src/test/java/com/google/cloud/datastore/execution/response/AggregationQueryResponseTransformerTest.java new file mode 100644 index 000000000..8776d4221 --- /dev/null +++ b/google-cloud-datastore/src/test/java/com/google/cloud/datastore/execution/response/AggregationQueryResponseTransformerTest.java @@ -0,0 +1,91 @@ +/* + * Copyright 2022 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.google.cloud.datastore.execution.response; + +import static com.google.cloud.datastore.ProtoTestData.intValue; +import static com.google.common.truth.Truth.assertThat; + +import com.google.cloud.Timestamp; +import com.google.cloud.datastore.AggregationResult; +import com.google.cloud.datastore.AggregationResults; +import com.google.cloud.datastore.LongValue; +import com.google.common.collect.ImmutableMap; +import com.google.datastore.v1.AggregationResultBatch; +import com.google.datastore.v1.RunAggregationQueryResponse; +import com.google.datastore.v1.Value; +import java.util.AbstractMap.SimpleEntry; +import java.util.HashMap; +import java.util.Map; +import java.util.Map.Entry; +import java.util.function.Function; +import java.util.stream.Collectors; +import org.junit.Test; + +public class AggregationQueryResponseTransformerTest { + + private final AggregationQueryResponseTransformer responseTransformer = + new AggregationQueryResponseTransformer(); + + @Test + public void shouldTransformAggregationQueryResponse() { + Map result1 = + new HashMap<>( + ImmutableMap.of( + "count", intValue(209), + "property_2", intValue(100))); + + Map result2 = + new HashMap<>( + ImmutableMap.of( + "count", intValue(509), + "property_2", intValue(100))); + Timestamp readTime = Timestamp.now(); + + AggregationResultBatch resultBatch = + AggregationResultBatch.newBuilder() + .addAggregationResults( + com.google.datastore.v1.AggregationResult.newBuilder() + .putAllAggregateProperties(result1) + .build()) + .addAggregationResults( + com.google.datastore.v1.AggregationResult.newBuilder() + .putAllAggregateProperties(result2) + .build()) + .setReadTime(readTime.toProto()) + .build(); + RunAggregationQueryResponse runAggregationQueryResponse = + RunAggregationQueryResponse.newBuilder().setBatch(resultBatch).build(); + + AggregationResults aggregationResults = + responseTransformer.transform(runAggregationQueryResponse); + + assertThat(aggregationResults.size()).isEqualTo(2); + assertThat(aggregationResults.get(0)).isEqualTo(new AggregationResult(toDomainValues(result1))); + assertThat(aggregationResults.get(1)).isEqualTo(new AggregationResult(toDomainValues(result2))); + assertThat(aggregationResults.getReadTime()).isEqualTo(readTime); + } + + private Map toDomainValues(Map map) { + + return map.entrySet().stream() + .map( + (Function, Entry>) + entry -> + new SimpleEntry<>( + entry.getKey(), (LongValue) LongValue.fromPb(entry.getValue()))) + .collect(Collectors.toMap(Entry::getKey, Entry::getValue)); + } +} diff --git a/google-cloud-datastore/src/test/java/com/google/cloud/datastore/it/ITDatastoreTest.java b/google-cloud-datastore/src/test/java/com/google/cloud/datastore/it/ITDatastoreTest.java index 869984932..b8c3bb4b6 100644 --- a/google-cloud-datastore/src/test/java/com/google/cloud/datastore/it/ITDatastoreTest.java +++ b/google-cloud-datastore/src/test/java/com/google/cloud/datastore/it/ITDatastoreTest.java @@ -16,6 +16,9 @@ package com.google.cloud.datastore.it; +import static com.google.cloud.datastore.aggregation.Aggregation.count; +import static com.google.common.collect.Iterables.getOnlyElement; +import static com.google.common.truth.Truth.assertThat; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; import static org.junit.Assert.assertNotEquals; @@ -26,14 +29,17 @@ import static org.junit.Assert.fail; import com.google.cloud.Timestamp; +import com.google.cloud.datastore.AggregationQuery; import com.google.cloud.datastore.Batch; import com.google.cloud.datastore.BooleanValue; import com.google.cloud.datastore.Cursor; import com.google.cloud.datastore.Datastore; +import com.google.cloud.datastore.Datastore.TransactionCallable; import com.google.cloud.datastore.DatastoreException; import com.google.cloud.datastore.DatastoreOptions; import com.google.cloud.datastore.DatastoreReaderWriter; import com.google.cloud.datastore.Entity; +import com.google.cloud.datastore.EntityQuery; import com.google.cloud.datastore.EntityValue; import com.google.cloud.datastore.FullEntity; import com.google.cloud.datastore.GqlQuery; @@ -61,13 +67,20 @@ import com.google.cloud.datastore.ValueType; import com.google.cloud.datastore.testing.RemoteDatastoreHelper; import com.google.common.base.Preconditions; +import com.google.common.collect.ImmutableList; import com.google.datastore.v1.TransactionOptions; +import com.google.datastore.v1.TransactionOptions.ReadOnly; import java.util.ArrayList; import java.util.Collections; import java.util.HashSet; import java.util.Iterator; import java.util.List; import java.util.Set; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.Future; +import java.util.function.BiConsumer; +import java.util.function.Consumer; import org.junit.After; import org.junit.AfterClass; import org.junit.Before; @@ -170,7 +183,11 @@ public void setUp() { @After public void tearDown() { - DATASTORE.delete(KEY1, KEY2, KEY3); + EntityQuery allEntitiesQuery = Query.newEntityQueryBuilder().build(); + QueryResults allEntities = DATASTORE.run(allEntitiesQuery); + Key[] keysToDelete = + ImmutableList.copyOf(allEntities).stream().map(Entity::getKey).toArray(Key[]::new); + DATASTORE.delete(keysToDelete); } private Iterator getStronglyConsistentResults(Query scQuery, Query query) @@ -506,6 +523,279 @@ public void testRunGqlQueryWithCasting() throws InterruptedException { assertFalse(results3.hasNext()); } + @Test + public void testRunAggregationQuery() { + // verifying aggregation with an entity query + testCountAggregationWith( + builder -> + builder + .addAggregation(count().as("total_count")) + .over( + Query.newEntityQueryBuilder().setNamespace(NAMESPACE).setKind(KIND1).build())); + + // verifying aggregation with a projection query + testCountAggregationWith( + builder -> + builder + .addAggregation(count().as("total_count")) + .over( + Query.newProjectionEntityQueryBuilder() + .setProjection("str") + .setNamespace(NAMESPACE) + .setKind(KIND1) + .build())); + + // verifying aggregation with a key query + testCountAggregationWith( + builder -> + builder + .addAggregation(count().as("total_count")) + .over(Query.newKeyQueryBuilder().setNamespace(NAMESPACE).setKind(KIND1).build())); + + // verifying aggregation with a GQL query + testCountAggregationWith( + builder -> + builder.over( + Query.newGqlQueryBuilder( + "AGGREGATE COUNT(*) AS total_count OVER (SELECT * FROM kind1)") + .setNamespace(NAMESPACE) + .build())); + } + + @Test + public void testRunAggregationQueryWithLimit() { + // verifying aggregation with an entity query + testCountAggregationWithLimit( + builder -> + builder + .addAggregation(count().as("total_count")) + .over(Query.newEntityQueryBuilder().setNamespace(NAMESPACE).setKind(KIND1).build()), + ((builder, limit) -> + builder + .addAggregation(count().as("total_count")) + .over( + Query.newEntityQueryBuilder() + .setNamespace(NAMESPACE) + .setKind(KIND1) + .setLimit(limit.intValue()) + .build()))); + + // verifying aggregation with a projection query + testCountAggregationWithLimit( + builder -> + builder + .addAggregation(count().as("total_count")) + .over( + Query.newProjectionEntityQueryBuilder() + .setProjection("str") + .setNamespace(NAMESPACE) + .setKind(KIND1) + .build()), + ((builder, limit) -> + builder + .addAggregation(count().as("total_count")) + .over( + Query.newProjectionEntityQueryBuilder() + .setProjection("str") + .setNamespace(NAMESPACE) + .setKind(KIND1) + .setLimit(limit.intValue()) + .build()))); + + // verifying aggregation with a key query + testCountAggregationWithLimit( + builder -> + builder + .addAggregation(count().as("total_count")) + .over(Query.newKeyQueryBuilder().setNamespace(NAMESPACE).setKind(KIND1).build()), + (builder, limit) -> + builder + .addAggregation(count().as("total_count")) + .over( + Query.newKeyQueryBuilder() + .setNamespace(NAMESPACE) + .setKind(KIND1) + .setLimit(limit.intValue()) + .build())); + + // verifying aggregation with a GQL query + testCountAggregationWithLimit( + builder -> + builder.over( + Query.newGqlQueryBuilder( + "AGGREGATE COUNT(*) AS total_count OVER (SELECT * FROM kind1)") + .setNamespace(NAMESPACE) + .build()), + (builder, limit) -> + builder.over( + Query.newGqlQueryBuilder( + "AGGREGATE COUNT(*) AS total_count OVER (SELECT * FROM kind1 LIMIT @limit)") + .setNamespace(NAMESPACE) + .setBinding("limit", limit) + .build())); + } + + /** + * if an entity is modified or deleted within a transaction, a query or lookup returns the + * original version of the entity as of the beginning of the transaction, or nothing if the entity + * did not exist then. + * + * @see + * Source + */ + @Test + public void testRunAggregationQueryInTransactionShouldReturnAConsistentSnapshot() { + Key newEntityKey = Key.newBuilder(KEY1, "newKind", "name-01").build(); + EntityQuery entityQuery = + Query.newEntityQueryBuilder() + .setNamespace(NAMESPACE) + .setFilter(PropertyFilter.hasAncestor(KEY1)) + .build(); + + AggregationQuery aggregationQuery = + Query.newAggregationQueryBuilder() + .setNamespace(NAMESPACE) + .over(entityQuery) + .addAggregation(count().as("count")) + .build(); + + // original entity count is 2 + assertThat(getOnlyElement(DATASTORE.runAggregation(aggregationQuery)).get("count")) + .isEqualTo(2L); + + // FIRST TRANSACTION + DATASTORE.runInTransaction( + (TransactionCallable) + inFirstTransaction -> { + // creating a new entity + Entity aNewEntity = + Entity.newBuilder(ENTITY2).setKey(newEntityKey).set("v_int", 10).build(); + inFirstTransaction.put(aNewEntity); + + // count remains 2 + assertThat( + getOnlyElement(inFirstTransaction.runAggregation(aggregationQuery)) + .get("count")) + .isEqualTo(2L); + assertThat(getOnlyElement(DATASTORE.runAggregation(aggregationQuery)).get("count")) + .isEqualTo(2L); + return null; + }); + // after first transaction is committed, count is updated to 3 now. + assertThat(getOnlyElement(DATASTORE.runAggregation(aggregationQuery)).get("count")) + .isEqualTo(3L); + + // SECOND TRANSACTION + DATASTORE.runInTransaction( + (TransactionCallable) + inSecondTransaction -> { + // deleting ENTITY2 + inSecondTransaction.delete(ENTITY2.getKey()); + + // count remains 3 + assertThat( + getOnlyElement(inSecondTransaction.runAggregation(aggregationQuery)) + .get("count")) + .isEqualTo(3L); + assertThat(getOnlyElement(DATASTORE.runAggregation(aggregationQuery)).get("count")) + .isEqualTo(3L); + return null; + }); + // after second transaction is committed, count is updated to 2 now. + assertThat(getOnlyElement(DATASTORE.runAggregation(aggregationQuery)).get("count")) + .isEqualTo(2L); + DATASTORE.delete(newEntityKey); + } + + @Test + public void testRunAggregationQueryInAReadOnlyTransactionShouldNotLockTheCountedDocuments() + throws Exception { + ExecutorService executor = Executors.newSingleThreadExecutor(); + EntityQuery entityQuery = + Query.newEntityQueryBuilder() + .setNamespace(NAMESPACE) + .setFilter(PropertyFilter.hasAncestor(KEY1)) + .build(); + AggregationQuery aggregationQuery = + Query.newAggregationQueryBuilder() + .setNamespace(NAMESPACE) + .over(entityQuery) + .addAggregation(count().as("count")) + .build(); + + TransactionOptions transactionOptions = + TransactionOptions.newBuilder().setReadOnly(ReadOnly.newBuilder().build()).build(); + Transaction readOnlyTransaction = DATASTORE.newTransaction(transactionOptions); + + // Executing query in transaction + assertThat(getOnlyElement(readOnlyTransaction.runAggregation(aggregationQuery)).get("count")) + .isEqualTo(2L); + + // Concurrent write task. + Future addNewEntityTaskOutsideTransaction = + executor.submit( + () -> { + Entity aNewEntity = + Entity.newBuilder(ENTITY2) + .setKey(Key.newBuilder(KEY1, "newKind", "name-01").build()) + .set("v_int", 10) + .build(); + DATASTORE.put(aNewEntity); + return null; + }); + + // should not throw exception and complete successfully as the ongoing transaction is read-only. + addNewEntityTaskOutsideTransaction.get(); + + // cleanup + readOnlyTransaction.commit(); + executor.shutdownNow(); + + assertThat(getOnlyElement(DATASTORE.runAggregation(aggregationQuery)).get("count")) + .isEqualTo(3L); + } + + @Test + public void testRunAggregationQueryWithReadTime() throws InterruptedException { + String alias = "total_count"; + + // verifying aggregation readTime with an entity query + testCountAggregationReadTimeWith( + builder -> + builder + .over(Query.newEntityQueryBuilder().setKind("new_kind").build()) + .addAggregation(count().as(alias))); + + // verifying aggregation readTime with a projection query + testCountAggregationReadTimeWith( + builder -> + builder + .over( + Query.newProjectionEntityQueryBuilder() + .setProjection("name") + .setKind("new_kind") + .build()) + .addAggregation(count().as(alias))); + + // verifying aggregation readTime with a key query + testCountAggregationReadTimeWith( + builder -> + builder + .over(Query.newKeyQueryBuilder().setKind("new_kind").build()) + .addAggregation(count().as(alias))); + + // verifying aggregation readTime with a GQL query + testCountAggregationReadTimeWith( + builder -> + builder + .over( + Query.newGqlQueryBuilder( + "AGGREGATE COUNT(*) AS total_count OVER (SELECT * FROM new_kind)") + .build()) + .addAggregation(count().as(alias))); + } + @Test public void testRunStructuredQuery() throws InterruptedException { Query query = @@ -1067,4 +1357,92 @@ public void testQueryWithReadTime() throws InterruptedException { DATASTORE.delete(entity1.getKey(), entity2.getKey(), entity3.getKey()); } } + + private void testCountAggregationWith(Consumer configurer) { + AggregationQuery.Builder builder = Query.newAggregationQueryBuilder().setNamespace(NAMESPACE); + configurer.accept(builder); + AggregationQuery aggregationQuery = builder.build(); + String alias = "total_count"; + + Long countBeforeAdd = getOnlyElement(DATASTORE.runAggregation(aggregationQuery)).get(alias); + long expectedCount = countBeforeAdd + 1; + + Entity newEntity = + Entity.newBuilder(ENTITY1) + .setKey(Key.newBuilder(KEY3, KIND1, 1).build()) + .set("null", NULL_VALUE) + .set("partial1", PARTIAL_ENTITY2) + .set("partial2", ENTITY2) + .build(); + DATASTORE.put(newEntity); + + Long countAfterAdd = getOnlyElement(DATASTORE.runAggregation(aggregationQuery)).get(alias); + assertThat(countAfterAdd).isEqualTo(expectedCount); + + DATASTORE.delete(newEntity.getKey()); + } + + private void testCountAggregationWithLimit( + Consumer withoutLimitConfigurer, + BiConsumer withLimitConfigurer) { + String alias = "total_count"; + + AggregationQuery.Builder withoutLimitBuilder = + Query.newAggregationQueryBuilder().setNamespace(NAMESPACE); + withoutLimitConfigurer.accept(withoutLimitBuilder); + + Long currentCount = + getOnlyElement(DATASTORE.runAggregation(withoutLimitBuilder.build())).get(alias); + long limit = currentCount - 1; + + AggregationQuery.Builder withLimitBuilder = + Query.newAggregationQueryBuilder().setNamespace(NAMESPACE); + withLimitConfigurer.accept(withLimitBuilder, limit); + + Long countWithLimit = + getOnlyElement(DATASTORE.runAggregation(withLimitBuilder.build())).get(alias); + assertThat(countWithLimit).isEqualTo(limit); + } + + private void testCountAggregationReadTimeWith(Consumer configurer) + throws InterruptedException { + Entity entity1 = + Entity.newBuilder( + Key.newBuilder(PROJECT_ID, "new_kind", "name-01").setNamespace(NAMESPACE).build()) + .set("name", "Tyrion Lannister") + .build(); + Entity entity2 = + Entity.newBuilder( + Key.newBuilder(PROJECT_ID, "new_kind", "name-02").setNamespace(NAMESPACE).build()) + .set("name", "Jaime Lannister") + .build(); + Entity entity3 = + Entity.newBuilder( + Key.newBuilder(PROJECT_ID, "new_kind", "name-03").setNamespace(NAMESPACE).build()) + .set("name", "Cersei Lannister") + .build(); + + DATASTORE.put(entity1, entity2); + Thread.sleep(1000); + Timestamp now = Timestamp.now(); + Thread.sleep(1000); + DATASTORE.put(entity3); + + try { + AggregationQuery.Builder builder = Query.newAggregationQueryBuilder().setNamespace(NAMESPACE); + configurer.accept(builder); + AggregationQuery countAggregationQuery = builder.build(); + + Long latestCount = + getOnlyElement(DATASTORE.runAggregation(countAggregationQuery)).get("total_count"); + assertThat(latestCount).isEqualTo(3L); + + Long oldCount = + getOnlyElement(DATASTORE.runAggregation(countAggregationQuery, ReadOption.readTime(now))) + .get("total_count"); + assertThat(oldCount).isEqualTo(2L); + } finally { + DATASTORE.delete(entity1.getKey(), entity2.getKey(), entity3.getKey()); + } + } } From 76a187a48cdc8d40fa233123894a36e11c590ca9 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Mon, 17 Oct 2022 17:37:42 +0200 Subject: [PATCH 08/11] deps: update dependency org.graalvm.buildtools:native-maven-plugin to v0.9.15 (#879) --- samples/native-image-sample/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/samples/native-image-sample/pom.xml b/samples/native-image-sample/pom.xml index aab2b9174..fdaa21a28 100644 --- a/samples/native-image-sample/pom.xml +++ b/samples/native-image-sample/pom.xml @@ -107,7 +107,7 @@ org.graalvm.buildtools native-maven-plugin - 0.9.14 + 0.9.15 true com.example.datastore.NativeImageDatastoreSample From ed816e20b605882a9cf2c637145597fdcd95f324 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Mon, 17 Oct 2022 17:39:46 +0200 Subject: [PATCH 09/11] deps: update dependency org.easymock:easymock to v5 (#877) --- pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pom.xml b/pom.xml index 3ed672f27..3324a2f52 100644 --- a/pom.xml +++ b/pom.xml @@ -197,7 +197,7 @@ org.easymock easymock - 4.3 + 5.0.0 test From 831a92bdc1d3f81fb44ae8d17cad236a50234ea5 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Mon, 17 Oct 2022 17:41:54 +0200 Subject: [PATCH 10/11] deps: update dependency org.graalvm.buildtools:junit-platform-native to v0.9.15 (#878) --- samples/native-image-sample/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/samples/native-image-sample/pom.xml b/samples/native-image-sample/pom.xml index fdaa21a28..24d9d49f0 100644 --- a/samples/native-image-sample/pom.xml +++ b/samples/native-image-sample/pom.xml @@ -86,7 +86,7 @@ org.graalvm.buildtools junit-platform-native - 0.9.14 + 0.9.15 test From 1f2dec1e7f79768dbb1bd3aacaff27675a9c21f5 Mon Sep 17 00:00:00 2001 From: "release-please[bot]" <55107282+release-please[bot]@users.noreply.github.com> Date: Mon, 17 Oct 2022 12:15:44 -0400 Subject: [PATCH 11/11] chore(main): release 2.12.0 (#874) Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com> --- CHANGELOG.md | 15 +++++++++++++++ datastore-v1-proto-client/pom.xml | 4 ++-- google-cloud-datastore-bom/pom.xml | 10 +++++----- google-cloud-datastore/pom.xml | 4 ++-- grpc-google-cloud-datastore-admin-v1/pom.xml | 4 ++-- pom.xml | 12 ++++++------ proto-google-cloud-datastore-admin-v1/pom.xml | 4 ++-- proto-google-cloud-datastore-v1/pom.xml | 4 ++-- versions.txt | 12 ++++++------ 9 files changed, 42 insertions(+), 27 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3746743ed..52908cfb9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,20 @@ # Changelog +## [2.12.0](https://github.com/googleapis/java-datastore/compare/v2.11.5...v2.12.0) (2022-10-17) + + +### Features + +* Count API ([#823](https://github.com/googleapis/java-datastore/issues/823)) ([8c22e61](https://github.com/googleapis/java-datastore/commit/8c22e61f8a0307a59301259f83a16c8324fa1b6f)) + + +### Dependencies + +* Update dependency com.google.errorprone:error_prone_core to v2.16 ([#872](https://github.com/googleapis/java-datastore/issues/872)) ([b2a72ca](https://github.com/googleapis/java-datastore/commit/b2a72ca407b1fa168c18b136e73932c8716fbdf6)) +* Update dependency org.easymock:easymock to v5 ([#877](https://github.com/googleapis/java-datastore/issues/877)) ([ed816e2](https://github.com/googleapis/java-datastore/commit/ed816e20b605882a9cf2c637145597fdcd95f324)) +* Update dependency org.graalvm.buildtools:junit-platform-native to v0.9.15 ([#878](https://github.com/googleapis/java-datastore/issues/878)) ([831a92b](https://github.com/googleapis/java-datastore/commit/831a92bdc1d3f81fb44ae8d17cad236a50234ea5)) +* Update dependency org.graalvm.buildtools:native-maven-plugin to v0.9.15 ([#879](https://github.com/googleapis/java-datastore/issues/879)) ([76a187a](https://github.com/googleapis/java-datastore/commit/76a187a48cdc8d40fa233123894a36e11c590ca9)) + ## [2.11.5](https://github.com/googleapis/java-datastore/compare/v2.11.4...v2.11.5) (2022-10-03) diff --git a/datastore-v1-proto-client/pom.xml b/datastore-v1-proto-client/pom.xml index 028c2d459..36e26b874 100644 --- a/datastore-v1-proto-client/pom.xml +++ b/datastore-v1-proto-client/pom.xml @@ -19,12 +19,12 @@ 4.0.0 com.google.cloud.datastore datastore-v1-proto-client - 2.11.6-SNAPSHOT + 2.12.0 com.google.cloud google-cloud-datastore-parent - 2.11.6-SNAPSHOT + 2.12.0 jar diff --git a/google-cloud-datastore-bom/pom.xml b/google-cloud-datastore-bom/pom.xml index 74d2a04e1..e9c9428a0 100644 --- a/google-cloud-datastore-bom/pom.xml +++ b/google-cloud-datastore-bom/pom.xml @@ -3,7 +3,7 @@ 4.0.0 com.google.cloud google-cloud-datastore-bom - 2.11.6-SNAPSHOT + 2.12.0 pom com.google.cloud @@ -52,22 +52,22 @@ com.google.cloud google-cloud-datastore - 2.11.6-SNAPSHOT + 2.12.0 com.google.api.grpc grpc-google-cloud-datastore-admin-v1 - 2.11.6-SNAPSHOT + 2.12.0 com.google.api.grpc proto-google-cloud-datastore-v1 - 0.102.6-SNAPSHOT + 0.103.0 com.google.api.grpc proto-google-cloud-datastore-admin-v1 - 2.11.6-SNAPSHOT + 2.12.0 diff --git a/google-cloud-datastore/pom.xml b/google-cloud-datastore/pom.xml index 27ca86ab5..9a61ffba0 100644 --- a/google-cloud-datastore/pom.xml +++ b/google-cloud-datastore/pom.xml @@ -2,7 +2,7 @@ 4.0.0 google-cloud-datastore - 2.11.6-SNAPSHOT + 2.12.0 jar Google Cloud Datastore https://github.com/googleapis/java-datastore @@ -12,7 +12,7 @@ com.google.cloud google-cloud-datastore-parent - 2.11.6-SNAPSHOT + 2.12.0 google-cloud-datastore diff --git a/grpc-google-cloud-datastore-admin-v1/pom.xml b/grpc-google-cloud-datastore-admin-v1/pom.xml index b6805ba68..21ca6f047 100644 --- a/grpc-google-cloud-datastore-admin-v1/pom.xml +++ b/grpc-google-cloud-datastore-admin-v1/pom.xml @@ -4,13 +4,13 @@ 4.0.0 com.google.api.grpc grpc-google-cloud-datastore-admin-v1 - 2.11.6-SNAPSHOT + 2.12.0 grpc-google-cloud-datastore-admin-v1 GRPC library for google-cloud-datastore com.google.cloud google-cloud-datastore-parent - 2.11.6-SNAPSHOT + 2.12.0 diff --git a/pom.xml b/pom.xml index 3324a2f52..e00b22d1f 100644 --- a/pom.xml +++ b/pom.xml @@ -4,7 +4,7 @@ com.google.cloud google-cloud-datastore-parent pom - 2.11.6-SNAPSHOT + 2.12.0 Google Cloud Datastore Parent https://github.com/googleapis/java-datastore @@ -159,27 +159,27 @@ com.google.api.grpc proto-google-cloud-datastore-admin-v1 - 2.11.6-SNAPSHOT + 2.12.0 com.google.api.grpc grpc-google-cloud-datastore-admin-v1 - 2.11.6-SNAPSHOT + 2.12.0 com.google.cloud google-cloud-datastore - 2.11.6-SNAPSHOT + 2.12.0 com.google.api.grpc proto-google-cloud-datastore-v1 - 0.102.6-SNAPSHOT + 0.103.0 com.google.cloud.datastore datastore-v1-proto-client - 2.11.6-SNAPSHOT + 2.12.0 com.google.api.grpc diff --git a/proto-google-cloud-datastore-admin-v1/pom.xml b/proto-google-cloud-datastore-admin-v1/pom.xml index 684d38fdf..2d84b01d6 100644 --- a/proto-google-cloud-datastore-admin-v1/pom.xml +++ b/proto-google-cloud-datastore-admin-v1/pom.xml @@ -4,13 +4,13 @@ 4.0.0 com.google.api.grpc proto-google-cloud-datastore-admin-v1 - 2.11.6-SNAPSHOT + 2.12.0 proto-google-cloud-datastore-admin-v1 Proto library for google-cloud-datastore com.google.cloud google-cloud-datastore-parent - 2.11.6-SNAPSHOT + 2.12.0 diff --git a/proto-google-cloud-datastore-v1/pom.xml b/proto-google-cloud-datastore-v1/pom.xml index 47a2cc7b9..9d78b2054 100644 --- a/proto-google-cloud-datastore-v1/pom.xml +++ b/proto-google-cloud-datastore-v1/pom.xml @@ -4,13 +4,13 @@ 4.0.0 com.google.api.grpc proto-google-cloud-datastore-v1 - 0.102.6-SNAPSHOT + 0.103.0 proto-google-cloud-datastore-v1 PROTO library for proto-google-cloud-datastore-v1 com.google.cloud google-cloud-datastore-parent - 2.11.6-SNAPSHOT + 2.12.0 diff --git a/versions.txt b/versions.txt index be559a86e..6c14878b5 100644 --- a/versions.txt +++ b/versions.txt @@ -1,9 +1,9 @@ # Format: # module:released-version:current-version -google-cloud-datastore:2.11.5:2.11.6-SNAPSHOT -google-cloud-datastore-bom:2.11.5:2.11.6-SNAPSHOT -proto-google-cloud-datastore-v1:0.102.5:0.102.6-SNAPSHOT -datastore-v1-proto-client:2.11.5:2.11.6-SNAPSHOT -proto-google-cloud-datastore-admin-v1:2.11.5:2.11.6-SNAPSHOT -grpc-google-cloud-datastore-admin-v1:2.11.5:2.11.6-SNAPSHOT +google-cloud-datastore:2.12.0:2.12.0 +google-cloud-datastore-bom:2.12.0:2.12.0 +proto-google-cloud-datastore-v1:0.103.0:0.103.0 +datastore-v1-proto-client:2.12.0:2.12.0 +proto-google-cloud-datastore-admin-v1:2.12.0:2.12.0 +grpc-google-cloud-datastore-admin-v1:2.12.0:2.12.0