Skip to content

Commit 7214ef6

Browse files
igorbernstein2kolea2
igorbernstein2
and
kolea2
authored
feat: Update opencensus metrics to include bigtable resource ids and rpc level metrics (#214)
* feat: Update opencensus metrics to include bigtable resource ids and rpc level metrics This PR refactors opencensus metrics integration to use gax's ApiTracers. Which allows this client to instrument individual attempts and tag everything with bigtable resource ids * fix deps * typo * remove unused param * add clirr * review feedback * code formatting * fix deps Co-authored-by: Kristen O'Leary <kaoleary@google.com>
1 parent 00f6d2d commit 7214ef6

21 files changed

Lines changed: 1236 additions & 1021 deletions

‎README.md‎

Lines changed: 32 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -176,13 +176,13 @@ If you are using Maven, add this to your pom.xml file
176176
<dependency>
177177
<groupId>io.opencensus</groupId>
178178
<artifactId>opencensus-impl</artifactId>
179-
<version>0.24.0</version>
179+
<version>0.26.0</version>
180180
<scope>runtime</scope>
181181
</dependency>
182182
<dependency>
183183
<groupId>io.opencensus</groupId>
184184
<artifactId>opencensus-exporter-trace-stackdriver</artifactId>
185-
<version>0.24.0</version>
185+
<version>0.26.0</version>
186186
<exclusions>
187187
<exclusion>
188188
<groupId>io.grpc</groupId>
@@ -197,13 +197,13 @@ If you are using Maven, add this to your pom.xml file
197197
```
198198
If you are using Gradle, add this to your dependencies
199199
```Groovy
200-
compile 'io.opencensus:opencensus-impl:0.24.0'
201-
compile 'io.opencensus:opencensus-exporter-trace-stackdriver:0.24.0'
200+
compile 'io.opencensus:opencensus-impl:0.26.0'
201+
compile 'io.opencensus:opencensus-exporter-trace-stackdriver:0.26.0'
202202
```
203203
If you are using SBT, add this to your dependencies
204204
```Scala
205-
libraryDependencies += "io.opencensus" % "opencensus-impl" % "0.24.0"
206-
libraryDependencies += "io.opencensus" % "opencensus-exporter-trace-stackdriver" % "0.24.0"
205+
libraryDependencies += "io.opencensus" % "opencensus-impl" % "0.26.0"
206+
libraryDependencies += "io.opencensus" % "opencensus-exporter-trace-stackdriver" % "0.26.0"
207207
```
208208

209209
At the start of your application configure the exporter:
@@ -236,30 +236,37 @@ Tracing.getTraceConfig().updateActiveTraceParams(
236236

237237
Cloud Bigtable client supports [Opencensus Metrics](https://opencensus.io/stats/),
238238
which gives insight into the client internals and aids in debugging production issues.
239-
Metrics prefixed with `cloud.google.com/java/bigtable/` focus on operation level
240-
metrics across all of the retry attempts that occurred during that operation. RPC
241-
level metrics can be gleaned from gRPC's metrics, which are prefixed with
242-
`grpc.io/client/`.
239+
All Cloud Bigtable Metrics are prefixed with `cloud.google.com/java/bigtable/`. The
240+
metrics will be tagged with:
241+
* `bigtable_project_id`: the project that contains the target Bigtable instance.
242+
Please note that this id could be different from project that the client is running
243+
in and different from the project where the metrics are exported to.
244+
* `bigtable_instance_id`: the instance id of the target Bigtable instance
245+
* `bigtable_app_profile_id`: the app profile id that is being used to access the target
246+
Bigtable instance
243247

244248
### Available operation level metric views:
245249

246-
* `cloud.google.com/java/bigtable/op_latency`: A distribution latency of
250+
* `cloud.google.com/java/bigtable/op_latency`: A distribution of latency of
247251
each client method call, across all of it's RPC attempts. Tagged by
248-
method name and final response status.
252+
operation name and final response status.
249253

250254
* `cloud.google.com/java/bigtable/completed_ops`: The total count of
251-
method invocations. Tagged by method name. Can be compared to
252-
`grpc.io/client/completed_rpcs` to visualize retry attempts.
255+
method invocations. Tagged by operation name and final response status.
253256

254257
* `cloud.google.com/java/bigtable/read_rows_first_row_latency`: A
255258
distribution of the latency of receiving the first row in a ReadRows
256259
operation.
257260

258-
* `cloud.google.com/java/bigtable/rows_per_op`: A distribution of rows
259-
read per ReadRows operation across all retry attempts.
261+
* `cloud.google.com/java/bigtable/attempt_latency`: A distribution of latency of
262+
each client RPC, tagged by operation name and the attempt status. Under normal
263+
circumstances, this will be identical to op_latency. However, when the client
264+
receives transient errors, op_latency will be the sum of all attempt_latencies
265+
and the exponential delays
260266

261-
* `cloud.google.com/java/bigtable/mutations_per_batch`: A distribution
262-
of mutations per BulkMutation.
267+
* `cloud.google.com/java/bigtable/attempts_per_op`: A distribution of attempts that
268+
each operation required, tagged by operation name and final operation status.
269+
Under normal circumstances, this will be 1.
263270

264271

265272
By default, the functionality is disabled. For example to enable metrics using
@@ -273,13 +280,13 @@ If you are using Maven, add this to your pom.xml file
273280
<dependency>
274281
<groupId>io.opencensus</groupId>
275282
<artifactId>opencensus-impl</artifactId>
276-
<version>0.24.0</version>
283+
<version>0.26.0</version>
277284
<scope>runtime</scope>
278285
</dependency>
279286
<dependency>
280287
<groupId>io.opencensus</groupId>
281288
<artifactId>opencensus-exporter-stats-stackdriver</artifactId>
282-
<version>0.24.0</version>
289+
<version>0.26.0</version>
283290
<exclusions>
284291
<exclusion>
285292
<groupId>io.grpc</groupId>
@@ -294,13 +301,13 @@ If you are using Maven, add this to your pom.xml file
294301
```
295302
If you are using Gradle, add this to your dependencies
296303
```Groovy
297-
compile 'io.opencensus:opencensus-impl:0.24.0'
298-
compile 'io.opencensus:opencensus-exporter-stats-stackdriver:0.24.0'
304+
compile 'io.opencensus:opencensus-impl:0.26.0'
305+
compile 'io.opencensus:opencensus-exporter-stats-stackdriver:0.26.0'
299306
```
300307
If you are using SBT, add this to your dependencies
301308
```Scala
302-
libraryDependencies += "io.opencensus" % "opencensus-impl" % "0.24.0"
303-
libraryDependencies += "io.opencensus" % "opencensus-exporter-stats-stackdriver" % "0.24.0"
309+
libraryDependencies += "io.opencensus" % "opencensus-impl" % "0.26.0"
310+
libraryDependencies += "io.opencensus" % "opencensus-exporter-stats-stackdriver" % "0.26.0"
304311
```
305312

306313
At the start of your application configure the exporter and enable the Bigtable stats views:
@@ -337,7 +344,7 @@ Add the following to your project's pom.xml.
337344
<dependency>
338345
<groupId>io.grpc</groupId>
339346
<artifactId>grpc-bom</artifactId>
340-
<version>1.27.0</version>
347+
<version>1.28.0</version>
341348
<type>pom</type>
342349
<scope>import</scope>
343350
</dependency>

‎google-cloud-bigtable-deps-bom/pom.xml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -190,6 +190,11 @@
190190
<artifactId>opencensus-api</artifactId>
191191
<version>${opencensus.version}</version>
192192
</dependency>
193+
<dependency>
194+
<groupId>io.opencensus</groupId>
195+
<artifactId>opencensus-impl-core</artifactId>
196+
<version>${opencensus.version}</version>
197+
</dependency>
193198
<dependency>
194199
<groupId>io.opencensus</groupId>
195200
<artifactId>opencensus-contrib-grpc-util</artifactId>

‎google-cloud-bigtable/clirr-ignored-differences.xml‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,4 +7,20 @@
77
<method>*snapshot*</method>
88
<to>*snapshot*</to>
99
</difference>
10+
<difference>
11+
<differenceType>8001</differenceType>
12+
<className>com/google/cloud/bigtable/data/v2/stub/metrics/MeasuredMutateRowsCallable*</className>
13+
</difference>
14+
<difference>
15+
<differenceType>8001</differenceType>
16+
<className>com/google/cloud/bigtable/data/v2/stub/metrics/MeasuredReadRowsCallable*</className>
17+
</difference>
18+
<difference>
19+
<differenceType>8001</differenceType>
20+
<className>com/google/cloud/bigtable/data/v2/stub/metrics/MeasuredUnaryCallable*</className>
21+
</difference>
22+
<difference>
23+
<differenceType>8001</differenceType>
24+
<className>com/google/cloud/bigtable/gaxx/tracing/WrappedTracerFactory*</className>
25+
</difference>
1026
</differences>

‎google-cloud-bigtable/pom.xml‎

Lines changed: 9 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -172,12 +172,6 @@
172172
<groupId>org.threeten</groupId>
173173
<artifactId>threetenbp</artifactId>
174174
</dependency>
175-
<!-- // TODO(igorbernstein): Enable grpc views once we upgrade to grpc-java 1.24.0 -->
176-
<!-- <dependency>-->
177-
<!-- <groupId>io.opencensus</groupId>-->
178-
<!-- <artifactId>opencensus-contrib-grpc-metrics</artifactId>-->
179-
<!-- <version>${opencensus.version}</version>-->
180-
<!-- </dependency>-->
181175

182176
<!-- Test dependencies -->
183177
<dependency>
@@ -218,6 +212,11 @@
218212
<artifactId>grpc-testing</artifactId>
219213
<scope>test</scope>
220214
</dependency>
215+
<dependency>
216+
<groupId>io.opencensus</groupId>
217+
<artifactId>opencensus-impl</artifactId>
218+
<scope>test</scope>
219+
</dependency>
221220
<dependency>
222221
<groupId>junit</groupId>
223222
<artifactId>junit</artifactId>
@@ -385,8 +384,12 @@
385384
<!--
386385
grpc-grpclb is used at runtime using reflection
387386
grpc-auth is not directly used transitively, but is pulled to align with other grpc parts
387+
opencensus-impl-core is brought in transitively through opencensus-impl
388388
-->
389389
<usedDependencies>io.grpc:grpc-auth,io.grpc:grpc-grpclb,com.google.auto.value:auto-value</usedDependencies>
390+
<ignoredUsedUndeclaredDependencies>
391+
<ignoredUsedUndeclaredDependency>io.opencensus:opencensus-impl-core</ignoredUsedUndeclaredDependency>
392+
</ignoredUsedUndeclaredDependencies>
390393
</configuration>
391394
</plugin>
392395

‎google-cloud-bigtable/src/main/java/com/google/cloud/bigtable/data/v2/stub/EnhancedBigtableStub.java‎

Lines changed: 63 additions & 46 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,8 @@
2020
import com.google.api.gax.batching.Batcher;
2121
import com.google.api.gax.batching.BatcherImpl;
2222
import com.google.api.gax.core.BackgroundResource;
23+
import com.google.api.gax.core.GaxProperties;
24+
import com.google.api.gax.grpc.GaxGrpcProperties;
2325
import com.google.api.gax.grpc.GrpcCallSettings;
2426
import com.google.api.gax.grpc.GrpcRawCallableFactory;
2527
import com.google.api.gax.retrying.ExponentialRetryAlgorithm;
@@ -32,6 +34,7 @@
3234
import com.google.api.gax.rpc.ServerStreamingCallSettings;
3335
import com.google.api.gax.rpc.ServerStreamingCallable;
3436
import com.google.api.gax.rpc.UnaryCallable;
37+
import com.google.api.gax.tracing.OpencensusTracerFactory;
3538
import com.google.api.gax.tracing.SpanName;
3639
import com.google.api.gax.tracing.TracedServerStreamingCallable;
3740
import com.google.api.gax.tracing.TracedUnaryCallable;
@@ -59,9 +62,9 @@
5962
import com.google.cloud.bigtable.data.v2.models.RowAdapter;
6063
import com.google.cloud.bigtable.data.v2.models.RowMutation;
6164
import com.google.cloud.bigtable.data.v2.models.RowMutationEntry;
62-
import com.google.cloud.bigtable.data.v2.stub.metrics.MeasuredMutateRowsCallable;
63-
import com.google.cloud.bigtable.data.v2.stub.metrics.MeasuredReadRowsCallable;
64-
import com.google.cloud.bigtable.data.v2.stub.metrics.MeasuredUnaryCallable;
65+
import com.google.cloud.bigtable.data.v2.stub.metrics.CompositeTracerFactory;
66+
import com.google.cloud.bigtable.data.v2.stub.metrics.MetricsTracerFactory;
67+
import com.google.cloud.bigtable.data.v2.stub.metrics.RpcMeasureConstants;
6568
import com.google.cloud.bigtable.data.v2.stub.mutaterows.BulkMutateRowsUserFacingCallable;
6669
import com.google.cloud.bigtable.data.v2.stub.mutaterows.MutateRowsBatchingDescriptor;
6770
import com.google.cloud.bigtable.data.v2.stub.mutaterows.MutateRowsRetryingCallable;
@@ -73,10 +76,13 @@
7376
import com.google.cloud.bigtable.data.v2.stub.readrows.RowMergingCallable;
7477
import com.google.cloud.bigtable.gaxx.retrying.ApiResultRetryAlgorithm;
7578
import com.google.common.base.Preconditions;
79+
import com.google.common.collect.ImmutableList;
7680
import com.google.common.collect.ImmutableMap;
7781
import com.google.protobuf.ByteString;
7882
import io.opencensus.stats.Stats;
7983
import io.opencensus.stats.StatsRecorder;
84+
import io.opencensus.tags.TagKey;
85+
import io.opencensus.tags.TagValue;
8086
import io.opencensus.tags.Tagger;
8187
import io.opencensus.tags.Tags;
8288
import java.io.IOException;
@@ -98,16 +104,12 @@
98104
*/
99105
@InternalApi
100106
public class EnhancedBigtableStub implements AutoCloseable {
101-
private static final String TRACING_OUTER_CLIENT_NAME = "Bigtable";
107+
private static final String CLIENT_NAME = "Bigtable";
102108

103109
private final EnhancedBigtableStubSettings settings;
104110
private final ClientContext clientContext;
105111
private final RequestContext requestContext;
106112

107-
// TODO: This should probably move to ClientContext
108-
private final Tagger tagger;
109-
private final StatsRecorder statsRecorder;
110-
111113
private final ServerStreamingCallable<Query, Row> readRowsCallable;
112114
private final UnaryCallable<Query, Row> readRowCallable;
113115
private final UnaryCallable<String, List<KeyOffset>> sampleRowKeysCallable;
@@ -125,15 +127,58 @@ public static EnhancedBigtableStub create(EnhancedBigtableStubSettings settings)
125127
}
126128

127129
@InternalApi("Visible for testing")
128-
private EnhancedBigtableStub(
130+
public EnhancedBigtableStub(
129131
EnhancedBigtableStubSettings settings,
130132
ClientContext clientContext,
131133
Tagger tagger,
132134
StatsRecorder statsRecorder) {
133135
this.settings = settings;
134-
this.clientContext = clientContext;
135-
this.tagger = tagger;
136-
this.statsRecorder = statsRecorder;
136+
137+
this.clientContext =
138+
clientContext
139+
.toBuilder()
140+
.setTracerFactory(
141+
new CompositeTracerFactory(
142+
ImmutableList.of(
143+
// Add OpenCensus Tracing
144+
new OpencensusTracerFactory(
145+
ImmutableMap.<String, String>builder()
146+
// Annotate traces with the same tags as metrics
147+
.put(
148+
RpcMeasureConstants.BIGTABLE_PROJECT_ID.getName(),
149+
settings.getProjectId())
150+
.put(
151+
RpcMeasureConstants.BIGTABLE_INSTANCE_ID.getName(),
152+
settings.getInstanceId())
153+
.put(
154+
RpcMeasureConstants.BIGTABLE_APP_PROFILE_ID.getName(),
155+
settings.getAppProfileId())
156+
// Also annotate traces with library versions
157+
.put("gax", GaxGrpcProperties.getGaxGrpcVersion())
158+
.put("grpc", GaxGrpcProperties.getGrpcVersion())
159+
.put(
160+
"gapic",
161+
GaxProperties.getLibraryVersion(
162+
EnhancedBigtableStubSettings.class))
163+
.build()),
164+
// Add OpenCensus Metrics
165+
MetricsTracerFactory.create(
166+
tagger,
167+
statsRecorder,
168+
ImmutableMap.<TagKey, TagValue>builder()
169+
.put(
170+
RpcMeasureConstants.BIGTABLE_PROJECT_ID,
171+
TagValue.create(settings.getProjectId()))
172+
.put(
173+
RpcMeasureConstants.BIGTABLE_INSTANCE_ID,
174+
TagValue.create(settings.getInstanceId()))
175+
.put(
176+
RpcMeasureConstants.BIGTABLE_APP_PROFILE_ID,
177+
TagValue.create(settings.getAppProfileId()))
178+
.build()),
179+
// Add user configured tracer
180+
clientContext.getTracerFactory())))
181+
.build();
137182
this.requestContext =
138183
RequestContext.create(
139184
settings.getProjectId(), settings.getInstanceId(), settings.getAppProfileId());
@@ -196,17 +241,9 @@ public <RowT> ServerStreamingCallable<Query, RowT> createReadRowsCallable(
196241
new TracedServerStreamingCallable<>(
197242
readRowsUserCallable,
198243
clientContext.getTracerFactory(),
199-
SpanName.of(TRACING_OUTER_CLIENT_NAME, "ReadRows"));
200-
201-
ServerStreamingCallable<Query, RowT> measured =
202-
new MeasuredReadRowsCallable<>(
203-
traced,
204-
TRACING_OUTER_CLIENT_NAME + ".ReadRows",
205-
tagger,
206-
statsRecorder,
207-
clientContext.getClock());
244+
SpanName.of(CLIENT_NAME, "ReadRows"));
208245

209-
return measured.withDefaultCallContext(clientContext.getDefaultCallContext());
246+
return traced.withDefaultCallContext(clientContext.getDefaultCallContext());
210247
}
211248

212249
/**
@@ -393,19 +430,9 @@ private UnaryCallable<BulkMutation, Void> createBulkMutateRowsCallable() {
393430

394431
UnaryCallable<BulkMutation, Void> traced =
395432
new TracedUnaryCallable<>(
396-
userFacing,
397-
clientContext.getTracerFactory(),
398-
SpanName.of(TRACING_OUTER_CLIENT_NAME, "MutateRows"));
399-
400-
UnaryCallable<BulkMutation, Void> measured =
401-
new MeasuredMutateRowsCallable(
402-
traced,
403-
TRACING_OUTER_CLIENT_NAME + ".MutateRows",
404-
tagger,
405-
statsRecorder,
406-
clientContext.getClock());
433+
userFacing, clientContext.getTracerFactory(), SpanName.of(CLIENT_NAME, "MutateRows"));
407434

408-
return measured.withDefaultCallContext(clientContext.getDefaultCallContext());
435+
return traced.withDefaultCallContext(clientContext.getDefaultCallContext());
409436
}
410437

411438
/**
@@ -578,19 +605,9 @@ private <RequestT, ResponseT> UnaryCallable<RequestT, ResponseT> createUserFacin
578605

579606
UnaryCallable<RequestT, ResponseT> traced =
580607
new TracedUnaryCallable<>(
581-
inner,
582-
clientContext.getTracerFactory(),
583-
SpanName.of(TRACING_OUTER_CLIENT_NAME, methodName));
584-
585-
UnaryCallable<RequestT, ResponseT> measured =
586-
new MeasuredUnaryCallable<>(
587-
traced,
588-
TRACING_OUTER_CLIENT_NAME + "." + methodName,
589-
tagger,
590-
statsRecorder,
591-
clientContext.getClock());
608+
inner, clientContext.getTracerFactory(), SpanName.of(CLIENT_NAME, methodName));
592609

593-
return measured.withDefaultCallContext(clientContext.getDefaultCallContext());
610+
return traced.withDefaultCallContext(clientContext.getDefaultCallContext());
594611
}
595612
// </editor-fold>
596613

0 commit comments

Comments
 (0)