Skip to content

feat(knowledge): add token, sentence, recursive, and regex chunkers #4102

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
waleedlatif1 merged 20 commits into from
Apr 11, 2026
Merged

feat(knowledge): add token, sentence, recursive, and regex chunkers #4102

Changes from 1 commit
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
9f83f87
feat(knowledge): add token, sentence, recursive, and regex chunkers
waleedlatif1 Apr 11, 2026
59f86e9
fix(chunkers): standardize token estimation and use emcn dropdown
waleedlatif1 Apr 11, 2026
25abb8a
fix(chunkers): address research audit findings
waleedlatif1 Apr 11, 2026
211fe90
fix(chunkers): fix remaining audit issues across all chunkers
waleedlatif1 Apr 11, 2026
4872e75
chore(chunkers): lint formatting
waleedlatif1 Apr 11, 2026
fc006ee
updated styling
waleedlatif1 Apr 11, 2026
c5b9b2f
fix(chunkers): audit fixes and comprehensive tests
waleedlatif1 Apr 11, 2026
cb814ff
chore(chunkers): remove unnecessary comments and dead code
waleedlatif1 Apr 11, 2026
899fc68
fix(chunkers): address PR review comments
waleedlatif1 Apr 11, 2026
4c3508b
fix(chunkers): use consistent overlap pattern in regex fallback
waleedlatif1 Apr 11, 2026
3a26dad
fix(chunkers): prevent content loss in word boundary splitting
waleedlatif1 Apr 11, 2026
5e8b051
fix(chunkers): restore structured data token ratio and overlap joiner
waleedlatif1 Apr 11, 2026
a53f760
lint
waleedlatif1 Apr 11, 2026
ec6fa58
fix(chunkers): fall back to character-level overlap in sentence chunker
waleedlatif1 Apr 11, 2026
e391efa
fix(chunkers): fix log message and add missing month abbreviations
waleedlatif1 Apr 11, 2026
f7fe06a
lint
waleedlatif1 Apr 11, 2026
9c624db
fix(chunkers): restore structured data detection threshold to > 2
waleedlatif1 Apr 11, 2026
4fd7685
fix(chunkers): pass chunkOverlap to buildChunks in TokenChunker
waleedlatif1 Apr 11, 2026
97a0bd4
fix(chunkers): restore separator-as-joiner pattern in splitRecursively
waleedlatif1 Apr 11, 2026
2c5a852
feat(knowledge): add JSONL file support for knowledge base uploads
waleedlatif1 Apr 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix(chunkers): prevent content loss in word boundary splitting 
When splitAtWordBoundaries snaps end back to a word boundary, advance
pos from end (not pos + step) in non-overlapping mode. The step-based
advancement is preserved for the sliding window case (TokenChunker).
  • Loading branch information
@waleedlatif1
waleedlatif1 committed Apr 11, 2026
commit 3a26dad205e1f7a51aabab3be800202b908dfc53
14 changes: 10 additions & 4 deletions apps/sim/lib/chunkers/utils.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,6 @@ export function splitAtWordBoundaries(
chunkSizeChars: number,
stepChars?: number
): string[] {
const step = Math.max(1, stepChars ?? chunkSizeChars)
const parts: string[] = []
let pos = 0

Expand All @@ -79,9 +78,16 @@ export function splitAtWordBoundaries(
parts.push(part)
}

const nextPos = pos + step
if (nextPos >= text.length) break
pos = nextPos
if (stepChars !== undefined) {
// Sliding window: advance by step for predictable overlap
const nextPos = pos + Math.max(1, stepChars)
if (nextPos >= text.length) break
pos = nextPos
} else {
// Non-overlapping: advance from end of extracted content
if (end >= text.length) break
pos = end
}
while (pos < text.length && text[pos] === ' ') pos++
Comment thread
waleedlatif1 marked this conversation as resolved.
}

Expand Down
Loading