fix(knowledge): bound chunking separators so one config can't stall processing - #6735
Conversation
…rocessing `chunkingStrategyOptionsSchema.separators` accepted an arbitrary-length array of arbitrary-length strings, next to a `pattern` field already capped at 500 chars. `RecursiveChunker` splits the whole document once per separator and walks the list from the top for every oversized fragment, so a persisted config with thousands of non-matching separators cost seconds of synchronous CPU on every later document upload — work neither the processing `Promise.race` timeout nor the after-the-fact chunk-count cap can interrupt. Measured on a 21.3 MB document: 632 ms at 100 separators, 6.1 s at 1000, 36.8 s at 5000. - Bound `separators` to 32 entries of at most 100 characters on the write path. The largest built-in recipe (markdown) uses 16, so hand-tuned lists still fit. - Keep the stored/read shape tolerant, so a config written before the bound still lists instead of failing response validation. - Clamp in `RecursiveChunker` too, with a warning, so an already-persisted oversized list cannot reach the split loop. An over-long separator is dropped rather than truncated: a truncated separator matches where the configured one never did, silently re-cutting the document, while dropping it behaves like a separator that finds no match. A list left empty falls back to the recipe. - Walk non-matching separators iteratively instead of recursing, so stack depth no longer tracks the separator count. Verified behavior-preserving against the previous implementation over 4000 randomized configs — byte-identical output. - Validate in the create-base modal so the limit surfaces inline. After the fix the same 21.3 MB document costs ~300 ms at every separator count.
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
PR SummaryMedium Risk Overview API: New write-time Zod limits on UI: Create knowledge base modal validates the comma-separated separator field inline (shared Tests cover contracts, chunker clamp behavior, and backward-compatible reads. Reviewed by Cursor Bugbot for commit 6f94d58. Configure here. |
Greptile SummaryThis PR bounds custom recursive-chunking separators at API, UI, and runtime boundaries while preserving compatibility with legacy stored configurations.
Confidence Score: 5/5The PR appears safe to merge. No blocking failure remains, and the previously reported import-path issue is fixed at the current head.
|
| Filename | Overview |
|---|---|
| apps/sim/app/workspace/[workspaceId]/knowledge/components/create-base-modal/create-base-modal.tsx | Adds strategy-aware separator validation, shared parsing, and inline field errors. |
| apps/sim/lib/api/contracts/knowledge/base.ts | Separates bounded write validation from permissive legacy-config response validation. |
| apps/sim/lib/chunkers/constants.ts | Defines shared limits for separator count and per-separator length. |
| apps/sim/lib/chunkers/recursive-chunker.ts | Enforces runtime bounds and iteratively skips non-matching separators while retaining recipe fallback. |
| apps/sim/lib/api/contracts/knowledge/base.test.ts | Covers write rejection at both bounds and backward-compatible reads of legacy configurations. |
| apps/sim/lib/chunkers/recursive-chunker.test.ts | Covers list clamping, oversized-item removal, fallback behavior, and uses the required absolute imports. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart LR
UI[Create-base form] -->|Validate separator bounds| API[Knowledge-base write contract]
API -->|Persist bounded config| DB[(Knowledge-base config)]
Legacy[(Legacy unbounded config)] --> Read[Permissive read contract]
DB --> Read
Read --> Clamp[RecursiveChunker runtime clamp]
Clamp --> Iterate[Iterative separator traversal]
Iterate --> Chunks[Document chunks]
Clamp -->|No usable separators| Recipe[Built-in recipe fallback]
Recipe --> Chunks
Reviews (2): Last reviewed commit: "fix(knowledge): gate separator validatio..." | Re-trigger Greptile
- The separator refines ran for every strategy, but the field only renders for `recursive` and only that strategy submits it, so a value left behind by a strategy switch could block submit with no visible field to clear. Gated the same way the regex-pattern refine already is. - Use absolute imports in the chunker test, per the repo convention.
|
@cursor review |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 6f94d58. Configure here.
Summary
strategyOptions.separatorsat 32 entries of at most 100 characters on knowledge-base create/update. It sat next to apatternfield already capped at 500 chars, but had no bound of its own.RecursiveChunkersplits the whole document once per separator and restarts the list for every oversized fragment, so a saved config with thousands of non-matching separators burned seconds of synchronous CPU on every later document upload. The processingPromise.racetimeout and the chunk-count cap can't interrupt a synchronous split loop. On a 21.3 MB document: 632 ms at 100 separators, 6.1 s at 1000, 36.8 s at 5000 — now ~300 ms at every count.Type of Change
Testing
bun run check:audits(27 audits),bun run type-check, and 750 tests acrosslib/chunkers,lib/api/contracts/knowledge, andlib/knowledgepass.Checklist