fix(snapshots): Chunk image uploads to avoid fd exhaustion and 413 errors#3249
fix(snapshots): Chunk image uploads to avoid fd exhaustion and 413 errors#3249NicoHinderling merged 2 commits intomasterfrom
Conversation
|
|
||
| const IMAGE_EXTENSIONS: &[&str] = &["png", "jpg", "jpeg"]; | ||
| const MAX_PIXELS_PER_IMAGE: u64 = 40_000_000; | ||
| const UPLOAD_BATCH_SIZE: usize = 100; |
There was a problem hiding this comment.
open to your guys' thoughts, but through some testing, seems to be a good number
There was a problem hiding this comment.
Once these two fixes are available, the custom batching can be removed again. Objectstore client batches internally:
This stack of pull requests is managed by Graphite. Learn more about stacking. |
3b2f5fa to
938a884
Compare
938a884 to
835e271
Compare
|
synced offline with @lcian , putting in draft mode since while fd limit is real, we on client side apparently shouldn't be handling the batching |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
|
Updated objectstore-client to 0.1.4. The fd limit is resolved and we also limit requests to 100MB internally now in addition to the preexisting 1000 ops / request limit. Upload with a provided sample succeeds now. |
6333876 to
f30b441
Compare
Wrap upload errors in anyhow::Error before formatting so {:#} walks the
full cause chain instead of showing only the top-level message.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
cfdf12b to
ced473a
Compare
Uploading hundreds of images via
build snapshotsfails in two ways:Both failures were confirmed with a real 753-image upload. 734 out of 753 images failed with 413 errors.
This PR splits
upload_images()into two phases:session.many()builder, and send. This bounds both open file descriptors and HTTP request payload size.Also includes minor code quality improvements: simplified scope extraction, iterator-based collision formatting, and removal of redundant bindings.