Skip to content

Add MAXCOUNT and MAXSIZE options to XREAD and XREADGROUP#15282

Merged
sggeorgiev merged 12 commits into
redis:unstablefrom
sggeorgiev:xread-reply-limits
Jun 26, 2026
Merged

Add MAXCOUNT and MAXSIZE options to XREAD and XREADGROUP#15282
sggeorgiev merged 12 commits into
redis:unstablefrom
sggeorgiev:xread-reply-limits

Conversation

@sggeorgiev

@sggeorgiev sggeorgiev commented May 29, 2026

Copy link
Copy Markdown
Collaborator

Overview

This PR adds two new options, MAXCOUNT and MAXSIZE, to the XREAD and XREADGROUP stream commands. They cap the cumulative reply across all streams named in a single command: MAXCOUNT bounds the total number of entries returned, and MAXSIZE bounds the total reply size in bytes. Unlike the existing COUNT option — which limits entries on a per-stream basis — these new options apply a global budget for the whole command, giving clients a reliable way to bound a multi-stream read.

Problem Statement

XREAD/XREADGROUP accept multiple streams in one call, and COUNT only limits entries per stream. A read over N streams with COUNT C can therefore return up to N * C entries, and the total reply size is effectively unbounded — a single command can return a very large payload depending on how many streams match and how large each entry is.

For clients that need to bound the work and memory of a single round trip (e.g. fan-in consumers reading many streams, or latency-sensitive paths), there is no way today to say "give me at most K entries total" or "at most B bytes total" across all streams in one command. Working around this requires issuing per-stream reads and aggregating client-side, costing extra round trips.

Solution

Introduces two cumulative caps, applied across all streams in the command:

  • XREAD [COUNT count] [MAXCOUNT maxcount] [MAXSIZE maxsize] [BLOCK milliseconds] STREAMS key [key ...] id [id ...]
  • XREADGROUP GROUP group consumer [COUNT count] [MAXCOUNT maxcount] [MAXSIZE maxsize] [BLOCK milliseconds] [CLAIM min-idle-time] [NOACK] STREAMS key [key ...] id [id ...]

Semantics:

  • MAXCOUNT caps the total number of entries returned across all streams. It must be a positive integer, and must be >= COUNT when both are given (since it is a cumulative cap over a per-stream limit). When COUNT is omitted, MAXCOUNT alone bounds the total.
  • MAXSIZE caps the total reply size in bytes (tracked via c->net_output_bytes_curr_cmd). It must be a positive integer.
  • At least one entry is always returned. The budget is never enforced until at least one entry has been emitted across the whole reply, so a single message larger than MAXSIZE is still returned rather than yielding an empty reply.
  • Both caps work for new (>) reads and for history/PEL reads in XREADGROUP. For MAXSIZE, the limit is checked before delivering the next PEL/new entry, so a skipped entry is neither sent nor added to the consumer's PEL.

Implementation details:

MAXCOUNT is enforced in xreadCommand(): a running total_entries counter shrinks each stream's effective count to the remaining budget and stops scanning further streams once the cap is reached. MAXSIZE is enforced inside the emit loops of streamReplyWithRange() and streamReplyWithRangeFromConsumerPEL(), guarded by (emitted_before + arraylen) > 0 to preserve the "always serve one entry" exception. All existing callers (XRANGE, XCLAIM, XAUTOCLAIM, XINFO, and the consumer-PEL path) are migrated to the new struct form.

Examples

Given three streams, each with 100 entries:

MAXCOUNT caps the cumulative entry count (vs. per-stream COUNT):

> XREAD COUNT 50 STREAMS s1 s2 s3 0 0 0
# returns 150 entries total (50 per stream)

> XREAD COUNT 50 MAXCOUNT 80 STREAMS s1 s2 s3 0 0 0
# returns 80 entries total — capped across all streams

MAXCOUNT without COUNT — bounds the total directly, filling from the first stream onward:

> XREAD MAXCOUNT 7 STREAMS s1 s2 s3 0 0 0
# returns 7 entries total, all from s1

MAXSIZE bounds the reply size in bytes:

> XREAD MAXSIZE 200 STREAMS s1 s2 s3 0 0 0
# returns fewer entries than an unbounded read, but always >= 1

MAXSIZE always returns a single oversized message:

> XADD bigstream 1-1 f <5000-byte value>
> XREAD MAXSIZE 50 STREAMS bigstream 0
1) 1) "bigstream"
   2) 1) 1) "1-1"
         2) 1) "f"
            2) "<5000-byte value>"
# the single entry exceeds MAXSIZE but is still returned

Both together — whichever bound triggers first wins:

> XREAD MAXCOUNT 5 MAXSIZE 100000 STREAMS s1 s2 s3 0 0 0
# returns 5 entries (MAXCOUNT is the tighter bound)

XREADGROUP behaves identically for both new (>) reads and history/PEL reads, and the caps are honored after a blocking client is unblocked.

Backward Compatibility

This is a fully additive change. MAXCOUNT and MAXSIZE are new optional tokens recognized only within XREAD/XREADGROUP; when omitted, behavior is exactly as before. No existing command, argument, default, or reply format is modified, and no new command is introduced.


Note

Medium Risk
Touches core stream read/PEL delivery paths and consumer-group side effects when MAXSIZE stops before the next entry; behavior is additive but multi-stream reply shaping is easy to get wrong for clients.

Overview
XREAD and XREADGROUP gain optional MAXCOUNT and MAXSIZE (8.10.0) to cap the cumulative reply across all streams in one command—unlike per-stream COUNT.

MAXCOUNT limits total entries; xreadCommand tracks total_entries, shrinks each stream’s effective count, and stops scanning streams once the cap is hit. MAXSIZE limits reply bytes via c->net_output_bytes_curr_cmd; enforcement is in streamReplyWithRange() / PEL paths, with a baseline so MULTI/EXEC budgets apply per XREAD only. At least one entry is always returned (a single oversized message can exceed MAXSIZE). Validation rejects non-positive values and MAXCOUNT < COUNT.

streamReplyWithRange() now takes a streamReplyRangeArgs struct (including maxsize / emitted_before); existing callers are updated. Command metadata and Tcl tests cover validation, multi-stream caps, blocking unblock, and MULTI/EXEC behavior.

Reviewed by Cursor Bugbot for commit 30f52e1. Bugbot is set up for automated code reviews on this repo. Configure here.

Comment thread src/commands/xread.json
Comment thread src/stream.h Outdated
@sundb sundb added this to Redis 8.10 Jun 2, 2026
@sundb sundb added state:needs-doc-pr requires a PR to redis-doc repository release-notes indication that this issue needs to be mentioned in the release notes labels Jun 2, 2026
Comment thread src/t_stream.c
return;
}

/* MAXCOUNT is a cumulative cap across all streams, so it must not be

@LiorKogan LiorKogan Jun 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we also error if (maxcount && maxsize && maxcount > maxsize)?
This is most likely a user error, though it will cause no harm.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MAXCOUNT is in entries and MAXSIZE is in bytes — different units, so maxcount > maxsize is fine. They're independent limits, and whichever one is hit first wins. MAXCOUNT 1000 MAXSIZE 64 is perfectly valid.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is valid, but doesn't make much sense, since each entry requires at least one byte.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@LiorKogan I can think of another example. If the
user uses the following configuration.
set-max-listpack-entries is no longer effective.
Since the user wants to do this, I don't think it's appropriate to cause any inconvenience to the user.
This way, the user won't have to wonder why each entry requires at least one byte. Therefore, I cannot use this parameter in this way.

set-max-listpack-entries 128
set-max-listpack-value 1

@sggeorgiev
sggeorgiev marked this pull request as ready for review June 10, 2026 10:07
@fcostaoliveira

Copy link
Copy Markdown
Collaborator

This change touches performance-sensitive code paths. Adding the action:run-benchmark label will trigger the CE Performance suite so we can see the impact before merge.

Comment thread tests/unit/type/stream-cgroups.tcl Outdated
Comment thread tests/unit/type/stream-cgroups.tcl
Comment thread tests/unit/type/stream-cgroups.tcl Outdated
Comment thread src/t_stream.c
Comment thread src/t_stream.c Outdated
Co-authored-by: debing.sun <debing.sun@redis.com>
Comment thread src/t_stream.c Outdated
Comment thread src/t_stream.c Outdated
Comment thread src/t_stream.c Outdated
@sundb

sundb commented Jun 23, 2026

Copy link
Copy Markdown
Collaborator

net_output_bytes_curr_cmd is reset only in commandProcessed(), so et_output_bytes_curr_cmd was accumulated across the whole EXEC, that is to say, even with a very small stream entries, we can still reach the maxsize limit.

SET key 9999999 1
MULTI
GET key
XREAD MAXSIZE 500 STREAMS s 0
EXEC

Comment thread src/t_stream.c Outdated
Comment thread src/t_stream.c Outdated
Comment thread src/stream.h Outdated

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

Reviewed by Cursor Bugbot for commit 4ee7db8. Configure here.

Comment thread src/t_stream.c
Comment thread src/t_stream.c Outdated
Comment thread src/t_stream.c Outdated
sggeorgiev and others added 2 commits June 25, 2026 12:02
Co-authored-by: debing.sun <debing.sun@redis.com>
@sggeorgiev
sggeorgiev merged commit 08b465e into redis:unstable Jun 26, 2026
19 checks passed
@github-project-automation github-project-automation Bot moved this from Todo to Done in Redis 8.10 Jun 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release-notes indication that this issue needs to be mentioned in the release notes state:needs-doc-pr requires a PR to redis-doc repository

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

4 participants