fix: Pass a reader with seeking support to S3 SDK upload method#18360
Closed
erezrokah wants to merge 1 commit into
Closed
fix: Pass a reader with seeking support to S3 SDK upload method#18360erezrokah wants to merge 1 commit into
erezrokah wants to merge 1 commit into
Conversation
disq
approved these changes
Jun 24, 2024
disq
left a comment
Member
There was a problem hiding this comment.
CI failing because of separate repo, but LGTM
Member
Author
|
Replaced by #18361 |
kodiakhq Bot
pushed a commit
that referenced
this pull request
Jun 24, 2024
#### Summary Instead of #18360, below are debug logs I've added (in the S3 SDK) to measure the over allocation when uploading batches: ``` Allocated 5MB, needed 3.66MB Allocated 5MB, needed 616.00B Allocated 5MB, needed 1.11KB Allocated 5MB, needed 21.57KB Allocated 5MB, needed 849.00B Allocated 5MB, needed 1.09KB Allocated 5MB, needed 1.89KB Allocated 5MB, needed 4.81KB Allocated 5MB, needed 4.22KB Allocated 5MB, needed 16.42KB Allocated 5MB, needed 54.46KB Allocated 5MB, needed 3.98MB Allocated 5MB, needed 3.83KB Allocated 5MB, needed 302.00B Allocated 5MB, needed 1.39KB Allocated 5MB, needed 596.00B Allocated 5MB, needed 537.00B Allocated 5MB, needed 793.00B Allocated 5MB, needed 33.50KB Allocated 5MB, needed 392.00B Allocated 5MB, needed 310.00B Allocated 5MB, needed 721.00B Allocated 5MB, needed 2.04KB Allocated 5MB, needed 1.51KB Allocated 5MB, needed 64.49KB Allocated 5MB, needed 17.37KB Allocated 5MB, needed 1.92KB Allocated 5MB, needed 1.02KB Allocated 5MB, needed 25.58KB Allocated 5MB, needed 1.85MB Allocated 5MB, needed 16.24KB Allocated 5MB, needed 9.47KB Allocated 5MB, needed 1.21KB Allocated 5MB, needed 1.65KB Allocated 5MB, needed 1.40KB Allocated 5MB, needed 1.20KB Allocated 5MB, needed 926.00B Allocated 5MB, needed 5.98KB Allocated 5MB, needed 5.18KB Allocated 5MB, needed 10.39KB Allocated 5MB, needed 9.20KB ```
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
See comment in code for reasoning, I'll also open an issue for the AWS SDK with suggestions on how to improve it