Skip to content

POC Option 3: Internal Standalone Upload Client#13870

Draft
blakeli0 wants to merge 4 commits into
googleapis:mainfrom
blakeli0:feat/scotty-poc-option3
Draft

POC Option 3: Internal Standalone Upload Client#13870
blakeli0 wants to merge 4 commits into
googleapis:mainfrom
blakeli0:feat/scotty-poc-option3

Conversation

@blakeli0

Copy link
Copy Markdown
Contributor

Draft pull request showing the implementation of Option 3 (generic background HttpJsonResumableUploadClient delegation inside GrpcStub).

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for the Resumable Upload Protocol (RUP) in gax-java by adding ResumableUploadCallable and its HTTP/JSON transport implementation. The review feedback identifies several critical issues and areas for improvement in HttpJsonResumableUploadCall and HttpJsonResumableUploadCallable. These include potential resource leaks from an unclosed InputStream, potential ClassCastException risks when handling headers and transport channels, inefficient JSON re-serialization and repeated HttpRequestFactory instantiation, a potential integer overflow when calculating timeouts for distant deadlines, and improper handling of InterruptedException in the sleep helper.

attempt = 0;
long previousOffset = -1;

InputStream stream = uploadRequest.getStreamProvider().get();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The InputStream obtained from uploadRequest.getStreamProvider().get() is opened before the main upload loop but is never closed when the method exits (either normally by returning a response or abruptly by throwing an exception). This can lead to resource leaks (e.g., open file descriptors or network connections). Wrap the stream usage in a try-finally block to ensure it is always closed properly.

Comment on lines +367 to +376
for (Map.Entry<String, Object> entry : requestHeaders.getHeaders().entrySet()) {
String key = entry.getKey();
String value = (String) entry.getValue();

if (isMetadataHeaderDenylisted(key)) {
httpRequest.getHeaders().set("X-Goog-Upload-Header-" + key, value);
} else {
httpRequest.getHeaders().set(key, value);
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The header values in requestHeaders.getHeaders() are stored as Object and can be of type List<String> (especially for multi-valued headers or extra headers from the call context). Casting them directly to (String) will throw a ClassCastException at runtime. Instead, handle both List and single String values safely.

    for (Map.Entry<String, Object> entry : requestHeaders.getHeaders().entrySet()) {
      String key = entry.getKey();
      Object valueObj = entry.getValue();
      if (valueObj instanceof List) {
        for (Object val : (List<?>) valueObj) {
          if (val != null) {
            String value = val.toString();
            if (isMetadataHeaderDenylisted(key)) {
              httpRequest.getHeaders().set("X-Goog-Upload-Header-" + key, value);
            } else {
              httpRequest.getHeaders().set(key, value);
            }
          }
        }
      } else if (valueObj != null) {
        String value = valueObj.toString();
        if (isMetadataHeaderDenylisted(key)) {
          httpRequest.getHeaders().set("X-Goog-Upload-Header-" + key, value);
        } else {
          httpRequest.getHeaders().set(key, value);
        }
      }
    }

Comment on lines +681 to +687

private void sleep(final long ms) {
try {
Thread.sleep(ms);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

In Java, do not swallow InterruptedException. When catching it, restore the thread's interrupted status by calling Thread.currentThread().interrupt() and handle the interruption appropriately, such as by throwing a relevant exception to signal that the operation cannot proceed.

References
  1. In Java, do not swallow InterruptedException. When catching it, restore the thread's interrupted status by calling Thread.currentThread().interrupt() and handle the interruption appropriately, such as by throwing a relevant exception (e.g., SpannerException) to signal that the operation cannot proceed.

Comment on lines +73 to +74
HttpJsonTransportChannel transportChannel =
(HttpJsonTransportChannel) clientContext.getTransportChannel();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

If the clientContext passed to this callable is configured with a gRPC transport channel (which is the case when called from GrpcEchoStub if the background HTTP/JSON channel was not properly resolved), clientContext.getTransportChannel() will return a GrpcTransportChannel. Casting it directly to HttpJsonTransportChannel will throw a ClassCastException at runtime. Consider adding an explicit type check or ensuring that the background HTTP/JSON channel is fully resolved and wrapped before invoking the callable.

Comment on lines +344 to +351
if (!Strings.isNullOrEmpty(requestBody)) {
JSON_FACTORY.createJsonParser(requestBody).parse(tokenRequest);
initialContent =
new JsonHttpContent(JSON_FACTORY, tokenRequest)
.setMediaType(new HttpMediaType("application/json; charset=utf-8"));
} else {
initialContent = new EmptyContent();
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Parsing the already-serialized requestBody JSON string into a GenericData object only to re-serialize it using JsonHttpContent is highly inefficient. Since requestBody is already a valid JSON string, you can use ByteArrayContent.fromString directly to avoid this overhead.

    if (!Strings.isNullOrEmpty(requestBody)) {
      initialContent = 
          ByteArrayContent.fromString("application/json; charset=utf-8", requestBody);
    } else {
      initialContent = new EmptyContent();
    }

Comment on lines +699 to +705
private HttpRequestFactory getRequestFactory() {
Credentials credentials = callOptions.getCredentials();
if (credentials != null) {
return httpTransport.createRequestFactory(new HttpCredentialsAdapter(credentials));
}
return httpTransport.createRequestFactory();
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Creating a new HttpRequestFactory for every single request/chunk is inefficient because it sets up the credentials adapter and initializer repeatedly. Since HttpRequestFactory is thread-safe and reusable, it should be initialized once in the constructor and stored as a final field.

Comment on lines +623 to +630
private void configureTimeouts(final HttpRequest request, final Instant deadline) {
long remainingMs = Duration.between(Instant.now(), deadline).toMillis();
if (remainingMs <= 0) {
remainingMs = 1; // force timeout
}
request.setConnectTimeout((int) remainingMs);
request.setReadTimeout((int) remainingMs);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If the configured deadline is very far in the future (e.g., more than 24 days), remainingMs can exceed Integer.MAX_VALUE. Casting it directly to (int) will cause an integer overflow, resulting in a negative timeout value which will throw an IllegalArgumentException or cause unexpected behavior. Clamp the value to Integer.MAX_VALUE to prevent this.

Suggested change
private void configureTimeouts(final HttpRequest request, final Instant deadline) {
long remainingMs = Duration.between(Instant.now(), deadline).toMillis();
if (remainingMs <= 0) {
remainingMs = 1; // force timeout
}
request.setConnectTimeout((int) remainingMs);
request.setReadTimeout((int) remainingMs);
}
private void configureTimeouts(final HttpRequest request, final Instant deadline) {
long remainingMs = Duration.between(Instant.now(), deadline).toMillis();
if (remainingMs <= 0) {
remainingMs = 1; // force timeout
}
int timeoutMs = (remainingMs > Integer.MAX_VALUE) ? Integer.MAX_VALUE : (int) remainingMs;
request.setConnectTimeout(timeoutMs);
request.setReadTimeout(timeoutMs);
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant