Skip to content

perf: Parallelize DynamoDB batch reads in sync online_read #6024

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ntkathole merged 6 commits into from
Mar 4, 2026
+368 −14
Merged

perf: Parallelize DynamoDB batch reads in sync online_read #6024

Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
6e65f4d
perf: Parallelize DynamoDB batch reads in sync online_read
abhijeet-dhumal Feb 25, 2026
c694ed9
test: add unit tests for DynamoDB parallel batch reads
abhijeet-dhumal Feb 25, 2026
e4a5202
fix: address thread-safety and max_workers issues in parallel DynamoD…
abhijeet-dhumal Feb 25, 2026
342587d
fix: Improve DynamoDB parallel reads: shared client, configurable wor…
abhijeet-dhumal Mar 2, 2026
2dce44b
docs: add max_read_workers config documentation for DynamoDB online s…
abhijeet-dhumal Mar 2, 2026
8ead314
docs: Fix default max worker count in docs
abhijeet-dhumal Mar 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
docs: Fix default max worker count in docs 
Signed-off-by: abhijeet-dhumal <abhijeetdhumal652@gmail.com>
  • Loading branch information
@abhijeet-dhumal
abhijeet-dhumal committed Mar 4, 2026
commit 8ead3149d1021785d2785d7d1f27b8853494b9c6
2 changes: 1 addition & 1 deletion docs/reference/online-stores/dynamodb.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ online_store:
| ------ | ---- | ------- | ----------- |
| `region` | string | | AWS region for DynamoDB |
| `table_name_template` | string | `{project}.{table_name}` | Template for table names |
| `batch_size` | int | `40` | Number of items per BatchGetItem/BatchWriteItem request (max 100) |
| `batch_size` | int | `100` | Number of items per BatchGetItem/BatchWriteItem request (max 100) |
| `max_read_workers` | int | `10` | Maximum parallel threads for batch read operations. Higher values improve throughput for large batch reads but increase resource usage |
| `consistent_reads` | bool | `false` | Whether to use strongly consistent reads (higher latency, guaranteed latest data) |
| `tags` | dict | `null` | AWS resource tags added to each table |
Expand Down
Loading