fix(knowledge): stop capping the unpaged knowledge-base list - #6771
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
PR SummaryMedium Risk Overview Copilot VFS rename/delete no longer loads the whole workspace with a substring search and filters in JS. They call new Removed unused constants from Reviewed by Cursor Bugbot for commit bc7d114. Configure here. |
Greptile SummaryThe PR removes fixed caps from unpaged knowledge-base and connector projections while preserving keyset limits for explicitly paged reads. It also replaces VFS name resolution through a workspace-wide search with a bounded exact-name query.
Confidence Score: 5/5The PR appears safe to merge because no blocking failure remains. No blocking failure remains.
|
| Filename | Overview |
|---|---|
| apps/sim/lib/knowledge/service.ts | Removes unpaged row and connector projection caps, preserves explicit paging, and adds a bounded exact-name lookup. |
| apps/sim/lib/knowledge/application/knowledge-vfs.ts | Switches VFS knowledge-base resolution from workspace list filtering to the exact-name service query. |
| apps/sim/lib/knowledge/service.test.ts | Replaces cap-failure coverage with unbounded-read, cursor paging, exact-name lookup, and oversized archived-list regression coverage. |
| apps/sim/lib/knowledge/constants.ts | Removes constants for the deleted knowledge-base and connector projection limits. |
Reviews (2): Last reviewed commit: "fix(knowledge): stop capping the unpaged..." | Re-trigger Greptile
#6770 routed GET /api/knowledge through the workspace read, which carried a 10,000-row cap. Staging crossed it, and the first list request after the deploy returned 500 with "Knowledge base list exceeds the 10000 row limit" — against data that had served fine for days. The cap could never have worked. Its throw was guarded by `limit === undefined`, so it fired only for callers that had NOT asked for a page: exactly the callers with no cursor to retry with and no way to ask for less. A paged caller never reached it. It also read one row PAST the cap before throwing, so it refused to serve rows it had already materialized — most of the memory was already spent. Soft-delete cleanup reclaims archived rows only past a retention window, and not at all where none is configured, so a workspace that archives faster than that window crosses any fixed count on its own. Remove it. An unpaged read is unbounded, matching the sibling internal lists (`listTables`, workspace files), and paged callers keep their page. That fixes the same latent 500 in the archived list, the catalog read, and the VFS name lookup, which are all unpaged too, rather than only the surface that failed. Two more of the same shape found while auditing for others: - `attachConnectorTypes` threw a bare Error above its own cap, on those same unpaged callers. Archiving a knowledge base archives its connectors, so the growth curve that broke staging could not reach it — but it is the identical construct, and the sibling `latestJobsForTables` has no equivalent. - The VFS path lookup read every knowledge base whose name merely CONTAINED the term and then exact-matched in JS, so a single-row lookup scaled with the workspace. It now queries the exact name and reads two rows, mirroring `findActiveTablesByExactName`.
0647e0b to
bc7d114
Compare
|
@cursor review |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit bc7d114. Configure here.
Summary
GET /api/knowledgereturned 500 withKnowledge base list exceeds the 10000 row limit. The same test had passed 28 consecutive runs over the prior two days against the same data — the delta was the code, not the volume.limit === undefined, so it fired only for callers that had not asked for a page — precisely the callers with no cursor to retry with and no way to ask for less. A paged caller never reached it. It also read one row past the cap before throwing, so it refused to serve rows it had already materialized; most of the memory cost was already paid.listTables, workspace files, neither capped). Paged callers keep their page and theirnextCursorKeys. This also clears the same latent 500 in three other unpaged callers — the archived list use case, the catalog read, and the VFS name lookup — rather than only the surface that happened to fail.Two more of the same shape, found while auditing for others:
attachConnectorTypesthrew a bareErrorabove its own row cap, on those same unpaged callers. Archiving a knowledge base archives its connectors, so the growth curve that broke staging could not reach it — but it is the identical construct, and the siblinglatestJobsForTableshas no equivalent.findActiveTablesByExactName.Regression test reproduces the incident: it fails with the cap restored, passes without.
Type of Change
Testing
Tested manually. Full
apps/simsuite,type-check,lint, and all 29 audits pass.Two things worth separate follow-up, neither in this PR:
sync-engine.tsaccumulates every synced document's extracted text into one array with a page-count bound but no byte cap. That is where a bound would actually earn its keep.Checklist