-
Notifications
You must be signed in to change notification settings - Fork 93
GCP cloud storage downloaded file corruption #2301
Copy link
Copy link
Labels
api: storageIssues related to the googleapis/java-storage API.Issues related to the googleapis/java-storage API.priority: p1Important issue which blocks shipping the next release. Will be fixed prior to next release.Important issue which blocks shipping the next release. Will be fixed prior to next release.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Metadata
Metadata
Assignees
Labels
api: storageIssues related to the googleapis/java-storage API.Issues related to the googleapis/java-storage API.priority: p1Important issue which blocks shipping the next release. Will be fixed prior to next release.Important issue which blocks shipping the next release. Will be fixed prior to next release.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Hi, recently we got an intermittent issue that the file size downloaded via storage sdk is different from the GCP cloud storage. Our initial investigation pointed us to here (code) that when an exception is thrown the retry won't update
positionthen data will be duplicated/corrupted.We wrote a simple test to verify.
And we set a breakpoint in

java.nio.channels.ChannelsWhen debugging this test and hitting this breakpoint, manually throw an

java.net.SocketTimeoutException. Then remove the breakpoint and Resume Program to let it proceed. And check the file size in local and bucket.I know this internal/hack way is not a perfect way to reproduce this issue, but it's just our first investigation and hard to reproduce externally.
Could this be a false alarm?
Thanks.