Add the text deriver, which embeds a segment's text without its trailing whitespace, as the default - #1857

Purpose of the change

The default deriver embeds a segment's text exactly as stored, including the whitespace at its end. That whitespace carries no content, and because the text is embedded as a JSON string, a trailing newline reaches the embedder as the two characters \n inside the quotes. This change adds a deriver that embeds the text without the whitespace at its end and makes it the default. The whitespace at a segment's start stays: it is the indentation of the segment's first line, which embedders use in code. The existing deriver stays, so configurations that name it keep working.

Description

• TextDeriver, configured as text, embeds text.rstrip(), keeping leading and inner whitespace, and gives a segment of whitespace alone no derivative. The name matches TextSegmenter (type: text) and says what the deriver reads, not how it prepares the text, so it stays true if that preparation changes.
• text becomes the event backend's default deriver, in its own commit.
• WholeTextDeriver (whole_text) is unchanged and still embeds the text exactly as stored. Whether to remove it is a separate decision.
• SentenceTextDeriver is unchanged; it already strips each sentence.
• The sample configurations name the new default and list both alternatives.

Embedded text. Changes for a configuration that names no deriver, wherever a message has whitespace at its end, for example a trailing newline: its derivative now ends at the last non-whitespace character. A message of whitespace alone no longer gets a derivative. Messages without trailing whitespace embed exactly as before, and a configuration that names whole_text is unaffected.

Fixes/Closes

No issue filed.

Type of change

• New feature (non-breaking change which adds functionality)

How Has This Been Tested?

• Unit Test

• test_text_deriver.py: TextDeriver embeds no trailing whitespace, keeps leading and inner whitespace, gives a whitespace-only segment no derivative, and keeps the segment's properties; WholeTextDeriver keeps edge whitespace.

• test_configuration.py: the default deriver is text, and type: text loads.

• test_service_locator.py: each deriver configuration builds its deriver.

Test Results:

• uv run --frozen ruff check packages/server and uv run --frozen ruff format --check packages/server: clean.
• uv run --frozen --all-extras ty check --project packages/server: clean.
• uv run --frozen --all-extras pytest -q -p no:cacheprovider on the three test files above: 58 passed (Python 3.14, macOS).

Checklist

• I have signed the commit(s) within this pull request
• My code follows the style guidelines of this project (See STYLE_GUIDE.md)
• I have performed a self-review of my own code
• I have commented my code
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added unit tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• Any dependent changes have been merged and published in downstream modules
• I have checked my code and corrected any misspellings

Maintainer Checklist

• Confirmed all checks passed
• Contributor has signed the commit(s)
• Reviewed the code
• Run, Tested, and Verified the change(s) work as expected

Screenshots/Gifs

N/A

Further comments

• Derivatives already stored keep their text; segments ingested after this change get trimmed derivatives.
• Split text blocks into segments that join back to the block, without langchain-text-splitters #1855 keeps the whitespace between a text block's segments in the segments themselves, so a split block renders exactly: the earlier segment ends with it through its last newline, and the later one starts with the rest, a line's indentation. With this change, the whitespace at a segment's end stays out of the embedded text, and its indentation stays in.
• Measured on code retrieval (802 docstring queries over 137 Python files, split by Split text blocks into segments that join back to the block, without langchain-text-splitters #1855 at 500 and 2,000 characters, MRR@10 with paired bootstrap): removing the whitespace at a segment's end lowered retrieval with none of text-embedding-3-small, embeddinggemma-300m, and Qwen3-Embedding-0.6B. Removing the indentation at its start lowered it with text-embedding-3-small at 500 characters (-0.9 points, 95% CI [-1.6, -0.1]) and with Qwen3-Embedding at both lengths.

🤖 Generated with Claude Code