Split text blocks into segments that join back to the block, without langchain-text-splitters - #1855

Purpose of the change

TextSegmenter's segments do not join back to the text they came from. It splits with LangChain's RecursiveCharacterTextSplitter, whose strip_whitespace defaults to True, so every chunk loses the whitespace at its ends. Event memory renders a run of segments by concatenating them, so a split block renders with its sentences run together. This change splits text in-house with the same separators, keeping every character, and places the whitespace between segments by a rule that LangChain's splitter cannot express. It drops langchain-text-splitters from the server's dependencies.

With max_chunk_length=50, "First sentence is here. Second sentence is right here. Third one ends it.\n\nNew paragraph starts here and goes on." segments as:

Rendered before: "First sentence is here.Second sentence is right here. Third one ends it.New paragraph starts here and goes on.". After: the text exactly. The second chunk splits because whitespace now counts toward the limit: with its trailing "\n\n", the old second chunk would be 51 code points.

Indentation stays with its line. With max_chunk_length=20, "def f():\n return [\n 1,\n 2,\n ]\n" segments as 'def f():\n return ', '[\n 1,\n', ' 2,\n ]\n'.

Description

Segments.

• A text block's segments, joined in offset order, are its text exactly.
• A whitespace run between two pieces of content is cut after its last newline: the earlier segment takes it through that newline, and the later one begins with the rest, the line's indentation. A run without a newline goes wholly to the earlier segment.
• A block's leading whitespace begins its first segment, and its trailing whitespace ends its last.
• No segment is longer than max_chunk_length, whitespace included. The content cuts are chosen so the rule's whitespace fits. A whitespace run that no segment can hold together with the character it attaches to is cut to fit instead; only then can a segment be whitespace alone.
• A block of whitespace alone is one segment when it fits; an empty block has none.
• A non-positive max_chunk_length raises ValueError, as LangChain's splitter did.

Why in-house rather than strip_whitespace=False.

• The rule decides where cuts may fall. LangChain's splitter offers no control over that, and moving whitespace after it splits would push chunks past the limit or leave whitespace on the wrong side.
• Upstream behavior has not changed. Stripping is in LangChain's earliest splitter: _join_docs calls text.strip() unconditionally at langchain v0.0.100 and v0.0.200, and strip_whitespace=True has been the default since stripping became optional. The functions on this segmenter's path (__init__, split_text, _split_text, _split_text_with_regex, _merge_splits, _join_docs) are identical in 1.1.1, 1.1.2, 1.1.3, and master, apart from an added @override decorator.
• It is still outside our control. The server declares langchain-text-splitters>=1.1.1 with no cap, so an install takes whatever release is current, and the chunk boundaries decide what is stored and what is embedded.
• Import weight. The locked 1.1.2 imports tiktoken and transformers at module import when they are installed. With all extras installed, import langchain_text_splitters took 1.3 s and loaded 2,451 modules on macOS. 1.1.3 makes those imports lazy again.
• Install size does not change much: langchain-aws keeps langchain-core in the base install.

Implementation. Three private functions in text_segmenter.py, covering only the configuration the segmenter uses: literal separators kept at the end of the text they follow, no overlap.

• _cuts finds the cut after each occurrence of a separator and moves any cut that touches whitespace to that run's cut, from _whitespace_cut, which scans only the run.
• _split recurses over the separators like LangChain's splitter, merging adjacent pieces up to max_chunk_length. Pieces already carry their whitespace when they merge, so no chunk exceeds the limit. At the last separator, the empty string, a piece still over the limit is cut every max_chunk_length code points.

Measured. 2,106 real documents (55.9 million characters from QASPER, COVID-QA, ConditionalQA, NQ, NarrativeQA, and LegalBench-RAG) and about 3,000 synthetic texts heavy in whitespace, blank-line runs, indentation, CJK punctuation, and code, at max_chunk_length 20, 128, and 500:

• the segments join to the text, and no segment exceeds max_chunk_length, in every case;
• whitespace runs too long to hold, so cut against the rule: on the real documents, none at 500, 1 at 128, and 57,621 at 20, where some documents hold whitespace runs longer than 20;
• segment count on the real documents at 500: 165,663, where moving whitespace after LangChain's split gives 165,548;
• speed: 0.86 s at 500 over the whole set; LangChain's splitter took 0.67 s on the same set.

Embedded text. Unchanged for the default configuration (passthrough segmenter, whole-text deriver). Where the text segmenter feeds the whole-text deriver, a derivative now carries its segment's edge whitespace inside the quoted text, as the passthrough segmenter's derivatives already do. The sentence deriver strips each sentence, so its derivatives are unchanged.

Dependencies. langchain-text-splitters leaves the server's dependencies. uv.lock was regenerated with uv 0.13.5, the release CI installs. It removes only the package's entries, and uv 0.13.5 also bumps the lock's revision from 3 to 5.

Fixes/Closes

No issue filed.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• Project Maintenance (updates to build scripts, CI, etc., that do not affect the main project)

How Has This Been Tested?

• Unit Test

New tests in segmenter/test_text_segmenter.py:

• segments join to the text, for sentences, edge whitespace, blank-line runs, indented code, CJK punctuation, an unbroken run, tabs and non-breaking spaces, and whitespace alone;
• 200 seeded random texts: segments join to the text, fit max_chunk_length, and every boundary that touches whitespace cuts it where the rule does;
• exact segments for the cut after a run's last newline, content cut so whitespace fits the limit, the separator preference, and leading and trailing whitespace;
• a whitespace run too long for any segment, cut to fit;
• whitespace-only and empty blocks, and a non-positive max_chunk_length.

In test_event_memory.py, a split event's queried context renders its text exactly. Against main's segmenter, 17 of the new tests fail; with this change all pass.

Test Results:

• uv lock --check (uv 0.13.5): passes.
• uv run --frozen ruff check packages/server and uv run --frozen ruff format --check packages/server: clean.
• uv run --frozen --all-extras ty check --project packages/server: clean.
• uv run --frozen --all-extras pytest packages/server/server_tests -q -p no:cacheprovider: 2,147 passed, 3 skipped, 1,725 deselected (Python 3.14, macOS).

Checklist

• I have signed the commit(s) within this pull request
• My code follows the style guidelines of this project (See STYLE_GUIDE.md)
• I have performed a self-review of my own code
• I have commented my code
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added unit tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• Any dependent changes have been merged and published in downstream modules
• I have checked my code and corrected any misspellings

Maintainer Checklist

• Confirmed all checks passed
• Contributor has signed the commit(s)
• Reviewed the code
• Run, Tested, and Verified the change(s) work as expected

Screenshots/Gifs

N/A

Further comments

• Segments already stored keep their stripped text; events ingested after this change get the new segments.
• A block of whitespace alone used to produce no segment. It now produces one, and under the whole-text deriver it gets a derivative of its quoted whitespace, as under the passthrough segmenter.
• The fork's agentic_expansion branch holds an earlier, verbatim port of the whole splitter (10ed25a6), which keeps LangChain's stripping and its unused options. This change replaces that approach.
• Keep the whitespace between a text block's segments #1856 keeps LangChain's splitter with strip_whitespace=False and moves whitespace after the split. It cannot keep indentation with its line or keep chunks within the limit.

🤖 Generated with Claude Code