fix(message-history): return the whole session from the messages property by ericdelorefice · Pull Request #752 · redis/redis-vl-python

ericdelorefice · 2026-10-13T14:11:45Z

Problem

SemanticMessageHistory.messages and MessageHistory.messages are both documented as "Returns the full message history." Neither returns it once a session gets long enough.

SemanticMessageHistory.messages builds its FilterQuery without num_results, so the FilterQuery default of 10 applies. A session with 12 messages comes back as its oldest 10. MessageHistory.messages sets num_results=1000 and stops there.

Both sort by timestamp ascending, so the messages that go missing are the most recent ones. Nothing raises or logs, and count() still reports the real total.

An app that rebuilds an LLM context, exports a transcript or summarises a conversation from messages loses everything after the 10th (semantic) or 1,000th (standard) message. get_recent(top_k=...) is not affected, because it passes top_k through as the query size.

Reproduction

Run against a local Redis 8.10.2 with the Query Engine on port 6390. The vectorizer is a stand-in, which is fine here because messages never uses vectors.

from redisvl.extensions.message_history import MessageHistory, SemanticMessageHistory
from redisvl.utils.vectorize import CustomVectorizer

url = "redis://localhost:6390"

semantic = SemanticMessageHistory(
    "demo_semantic",
    redis_url=url,
    overwrite=True,
    vectorizer=CustomVectorizer(lambda text: [0.1, 0.2, 0.3]),  # stand-in, never used here
)
semantic.add_messages([{"role": "user", "content": f"message {i}"} for i in range(12)])
print("semantic: count", semantic.count(), "messages", len(semantic.messages))

standard = MessageHistory("demo_standard", redis_url=url)
standard.add_messages([{"role": "user", "content": f"message {i}"} for i in range(1005)])
print("standard: count", standard.count(), "messages", len(standard.messages))

semantic.delete()
standard.delete()

On main:

semantic: count 12 messages 10
standard: count 1005 messages 1000

With this change:

semantic: count 12 messages 12
standard: count 1005 messages 1005

That is the full output of each run, nothing trimmed.

Change

Both properties now run one FilterQuery sized from count() and sorted by timestamp ascending, as @vishal-bala suggested in review (the first version paged with paginate, which re-sorts the session on every page). It is +12/-3 across the two files, docstrings included. The session filter and the sort are unchanged.

The docstrings now say messages returns every message in the current session, oldest first, with no bound, and point to get_recent(top_k=...) for a bounded read. An empty session gives num_results=0 and returns [].

The intended behaviour change is that a long session now returns every message, so the list can be large. Each read is 2 FT.SEARCH calls (the count, then the query) for both 12 and 1,005 messages, counted with INFO commandstats.

Limits worth knowing:

A message added between count() and the query is not included in that read.
A session larger than the server's search-max-search-results (1,000,000 here) now raises SEARCH_LIMIT_OVER instead of coming back short.
The semantic class's TODO about refactoring messages onto get_recent is left alone. This keeps the current API and only fixes the size.

Verification

Two new tests in tests/integration/test_message_history.py, against a real Redis:

test_standard_messages_property_past_1000: 1,005 messages, plus 15 in session_tag="other_session" that must not appear.
test_semantic_messages_property_past_10: 12 messages, using the existing semantic_history fixture and marked requires_hf.

Both fail on main (the standard one comes back 5 short, the semantic one 2 short) and pass with this change, 3 runs each way.

Local setup: Redis 8.10.2 with search-workers 0, Python 3.12.14 and the uv.lock pins; a small local pytest plugin pointed the compose fixture at that server, and is not part of this PR. sentence-transformers is not installed here, so for the semantic test a local stand-in replaced HFTextVectorizer with a fixed-size vectorizer (messages never uses vectors). The real requires_hf path will only run in CI.

On tests/integration/test_message_history.py without the stand-in, the only differences between main and this branch are the two new tests; the remaining failures and errors are the same both ways and come from the missing sentence-transformers. isort, black 26.5.1 and mypy ("no issues found in 120 source files") are clean on the changed files.

This is a bug fix, so the label would be auto:patch per CONTRIBUTING.md. A pull request from a fork cannot set labels.

A search of this repo's issues and PRs found no existing report of the cap.

This PR was written with AI assistance (Claude Code). Everything quoted above was run before filing.

Note

Medium Risk
Behavior change for long sessions (larger reads and two Redis searches per access); very large sessions may hit server search result limits instead of silent truncation.

Overview
Fixes a silent truncation bug where MessageHistory.messages and SemanticMessageHistory.messages did not return the full session. Standard history capped at 1,000 results; semantic history used FilterQuery's default of 10, so the newest messages were dropped while count() still reported the true total.

Both properties now set num_results=self.count() on the same timestamp-ascending FilterQuery, so reads return every message in the current session (oldest first). Docstrings clarify that messages is unbounded and point callers to get_recent(top_k=...) for a limited window.

Integration tests cover 1,005 standard messages (with a separate session tag excluded) and 12 semantic messages.

^{Reviewed by Cursor Bugbot for commit 7188c45. Bugbot is set up for automated code reviews on this repo. Configure here.}

…erty SemanticMessageHistory.messages built its FilterQuery without num_results, so the FilterQuery default of 10 applied and a longer session came back as its oldest 10 messages. MessageHistory.messages hard-coded num_results=1000 and stopped there. Both are documented as returning the full message history. Page through the session with SearchIndex.paginate instead, and add tests that fail on the old code (12 and 25 semantic messages, 1005 standard messages) plus controls for short and empty sessions, session scope and tool fields.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

vishal-bala

Thanks for your contribution and for finding this! Just a couple comments about the approach and then we can get this going

vishal-bala · 2026-10-13T15:15:58Z

-        messages = self._index.query(query)
+        # Page through the whole session: a single query stops at num_results.
+        messages = [
+            m for page in self._index.paginate(query, page_size=1000) for m in page


paginate re-sorts the whole session on every page, so the cost of a read grows with the square of the session length. On Redis 8.4, .messages took 1.7 s for a 50k-message session, against 0.3 s for one query. Sizing a single query from count() takes the same two round trips (paginate always ends with an empty page) and sorts once:

query = FilterQuery( filter_expression=self._default_session_filter, return_fields=return_fields, num_results=self.count(), ) query.sort_by(TIMESTAMP_FIELD_NAME, asc=True) messages = self._index.query(query)

It also sidesteps paginate's unique-sort-field requirement, which timestamps don't meet.

vishal-bala · 2026-10-13T15:15:58Z

-        messages = self._index.query(query)
+        # Page through the whole session: a single query stops at num_results.
+        messages = [
+            m for page in self._index.paginate(query, page_size=1000) for m in page


Same as above.

vishal-bala · 2026-10-13T15:15:58Z

        )
        query.sort_by(TIMESTAMP_FIELD_NAME, asc=True)
-        messages = self._index.query(query)
+        # Page through the whole session: a single query stops at num_results.


Could the messages docstring (here and in semantic_history.py) say this returns every message in the current session, oldest first, with no bound, and point to get_recent(top_k=...) for a bounded read?

vishal-bala · 2026-10-13T15:15:58Z



+@pytest.fixture
+def stub_semantic_history(app_name, client):


The existing semantic_history fixture already runs in CI and takes about 0.1 s for a 12-message test. Could past_10 use it with @pytest.mark.requires_hf, so this fixture and the CustomVectorizer import can go?

vishal-bala · 2026-10-13T15:15:58Z

+    assert standard_history.messages == messages
+
+
+def test_standard_messages_property_scope(standard_history):


This passes without the fix, and so does the semantic one. It still covers a real gap, though, since nothing else checks session scoping. Could you fold it into past_1000 with a single add_messages(..., session_tag="other_session") line and drop both standalone tests?

vishal-bala · 2026-10-13T15:15:58Z

+    assert stub_semantic_history.messages == messages
+
+
+def test_semantic_messages_property_matches_get_recent(stub_semantic_history):


past_10 already covers this, so let's drop it. The same goes for up_to_10 and keeps_tool_fields: both pass without the fix and repeat what test_semantic_messages_property already checks.

…r review Replace SearchIndex.paginate in both messages properties with a single FilterQuery sized by count(). paginate re-sorted the whole session for every page and needs a unique sort field, which timestamps are not. An empty session gives num_results=0 and returns []. Say in both docstrings that messages returns every message in the current session, oldest first, with no bound, and point to get_recent(top_k=...) for a bounded read. Tests: the 12-message semantic test uses the existing semantic_history fixture (requires_hf), so the stub fixture and CustomVectorizer import go. Session scoping is folded into the 1005-message test. Drop the get_recent comparison, which the 12-message test covers, and the standalone scope, up-to-10 and tool-field tests, which pass without the fix.

ericdelorefice · 2026-10-13T15:25:44Z

Thanks for the careful review, Vishal. I pushed a commit that addresses each point:

Both messages properties now run one FilterQuery sized from count() instead of paginate. An empty session gives num_results=0 and returns [].
Both docstrings now say it returns every message in the current session, oldest first, with no bound, and point to get_recent(top_k=...) for a bounded read.
The 12-message test uses the existing semantic_history fixture with requires_hf, so the stub fixture and CustomVectorizer import are gone.
Session scoping is one line in the 1,005-message test, and both scope tests are dropped.
Dropped the get_recent, up_to_10 and keeps_tool_fields tests.

sentence-transformers isn't installed on my machine, so I ran the semantic test with a local stand-in vectorizer; the real requires_hf path will run in CI. I also updated the PR description to match.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

vishal-bala

LGTM!

Copilot AI balanced review requested due to automatic review settings October 13, 2026 14:11

Copilot AI reviewed Oct 13, 2026

vishal-bala requested changes Oct 13, 2026

View reviewed changes

vishal-bala added the auto:patch Increment the patch version when merged label Oct 13, 2026

Copilot AI balanced review requested due to automatic review settings October 13, 2026 15:25

Copilot AI reviewed Oct 13, 2026

vishal-bala self-requested a review October 13, 2026 15:32

vishal-bala approved these changes Oct 14, 2026

View reviewed changes

vishal-bala merged commit fe583d5 into redis:main Oct 14, 2026
55 checks passed

Navigation Menu

Repository navigation

fix(message-history): return the whole session from the messages property - #752

fix(message-history): return the whole session from the messages property#752
vishal-bala merged 2 commits into
redis:mainfrom
ericdelorefice:fix/messages-property-full-history

ericdelorefice commented Oct 13, 2026 •

edited by cursor Bot

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

vishal-bala left a comment

Uh oh!

vishal-bala Oct 13, 2026

Uh oh!

vishal-bala Oct 13, 2026

Uh oh!

vishal-bala Oct 13, 2026

Uh oh!

vishal-bala Oct 13, 2026

Uh oh!

vishal-bala Oct 13, 2026

Uh oh!

vishal-bala Oct 13, 2026

Uh oh!

ericdelorefice commented Oct 13, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

vishal-bala left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		assert standard_history.messages == messages


		def test_standard_messages_property_scope(standard_history):

		assert stub_semantic_history.messages == messages


		def test_semantic_messages_property_matches_get_recent(stub_semantic_history):

Conversation

ericdelorefice commented Oct 13, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Reproduction

Change

Verification

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

vishal-bala left a comment

Choose a reason for hiding this comment

Uh oh!

vishal-bala Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

vishal-bala Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

vishal-bala Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

vishal-bala Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

vishal-bala Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

vishal-bala Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

ericdelorefice commented Oct 13, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

vishal-bala left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ericdelorefice commented Oct 13, 2026 •

edited by cursor Bot

Loading