I uploaded some big files with gsutil using gzip encoding:
gsutil cp -Z bigfile gs://<bucket_name>/
This code is OK when downloading the file:
Blob blob = storage.get(bucket, name);
try (ReadChannel reader = blob.reader()) {
int fileSize = blob.size().intValue();
reader.chunkSize(fileSize); //any bigger value is also OK
ByteStreams.copy(reader, Channels.newChannel(ByteStreams.nullOutputStream()));
}
This code fails (chunk size is smaller the content size):
Blob blob = storage.get(bucket, name);
try (ReadChannel reader = blob.reader()) {
int fileSize = blob.size().intValue();
reader.chunkSize(fileSize - 1);
ByteStreams.copy(reader, Channels.newChannel(ByteStreams.nullOutputStream()));
}
with this exception
com.google.cloud.storage.StorageException
at com.google.cloud.storage.spi.DefaultStorageRpc.translate(DefaultStorageRpc.java:98)
at com.google.cloud.storage.spi.DefaultStorageRpc.read(DefaultStorageRpc.java:474)
at com.google.cloud.storage.BlobReadChannel$1.call(BlobReadChannel.java:127)
at com.google.cloud.storage.BlobReadChannel$1.call(BlobReadChannel.java:124)
at com.google.cloud.RetryHelper.doRetry(RetryHelper.java:181)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:247)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:237)
at com.google.cloud.storage.BlobReadChannel.read(BlobReadChannel.java:124)
at com.google.common.io.ByteStreams.copy(ByteStreams.java:148)
at ...
Caused by: java.io.EOFException
at java.util.zip.GZIPInputStream.readUByte(GZIPInputStream.java:264)
at java.util.zip.GZIPInputStream.readUShort(GZIPInputStream.java:255)
at java.util.zip.GZIPInputStream.readUInt(GZIPInputStream.java:247)
at java.util.zip.GZIPInputStream.readTrailer(GZIPInputStream.java:218)
at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:118)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at com.google.api.client.util.ByteStreams.copy(ByteStreams.java:51)
at com.google.api.client.util.IOUtils.copy(IOUtils.java:94)
at com.google.api.client.util.IOUtils.copy(IOUtils.java:63)
at com.google.api.client.http.HttpResponse.download(HttpResponse.java:421)
at com.google.cloud.storage.spi.DefaultStorageRpc.read(DefaultStorageRpc.java:470)
... 31 more
It is of course not practical to increase the chunk size indefinitely, as the whole chunk is uncompressed in memory and might create OutOfMemoryErrors.
BTW, the generated Java API client fails with the same error (I suppose it uses the same http stack for downloads).
I uploaded some big files with gsutil using gzip encoding:
gsutil cp -Z bigfile gs://<bucket_name>/This code is OK when downloading the file:
This code fails (chunk size is smaller the content size):
with this exception
It is of course not practical to increase the chunk size indefinitely, as the whole chunk is uncompressed in memory and might create OutOfMemoryErrors.
BTW, the generated Java API client fails with the same error (I suppose it uses the same http stack for downloads).