Skip to content

fix(knowledge): stop capping the unpaged knowledge-base list - #6771

Merged
waleedlatif1 merged 1 commit into
stagingfrom
fix/kb-archived-list-cap
Aug 17, 2026
Merged

fix(knowledge): stop capping the unpaged knowledge-base list#6771
waleedlatif1 merged 1 commit into
stagingfrom
fix/kb-archived-list-cap

Conversation

@waleedlatif1

@waleedlatif1 waleedlatif1 commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Staging's integration test failed on the first run after fix(knowledge): list knowledge bases on the same authority that creates them #6770 deployed: GET /api/knowledge returned 500 with Knowledge base list exceeds the 10000 row limit. The same test had passed 28 consecutive runs over the prior two days against the same data — the delta was the code, not the volume.
  • fix(knowledge): list knowledge bases on the same authority that creates them #6770 routed the internal list through the workspace read, which carried a 10,000-row cap. Soft-delete cleanup reclaims archived knowledge bases only past a retention window, and not at all where none is configured, so a workspace that archives faster than that window crosses any fixed count on its own.
  • The cap could never have worked as a guard. Its throw was gated on limit === undefined, so it fired only for callers that had not asked for a page — precisely the callers with no cursor to retry with and no way to ask for less. A paged caller never reached it. It also read one row past the cap before throwing, so it refused to serve rows it had already materialized; most of the memory cost was already paid.
  • Removed. An unpaged read is unbounded, matching the sibling internal lists (listTables, workspace files, neither capped). Paged callers keep their page and their nextCursorKeys. This also clears the same latent 500 in three other unpaged callers — the archived list use case, the catalog read, and the VFS name lookup — rather than only the surface that happened to fail.

Two more of the same shape, found while auditing for others:

  • attachConnectorTypes threw a bare Error above its own row cap, on those same unpaged callers. Archiving a knowledge base archives its connectors, so the growth curve that broke staging could not reach it — but it is the identical construct, and the sibling latestJobsForTables has no equivalent.
  • The VFS path lookup read every knowledge base whose name merely contained the term and exact-matched in JS, so a single-row lookup scaled with the workspace. It now queries the exact name and reads two rows, mirroring findActiveTablesByExactName.

Regression test reproduces the incident: it fails with the cap restored, passes without.

Type of Change

  • Bug fix

Testing

Tested manually. Full apps/sim suite, type-check, lint, and all 29 audits pass.

Two things worth separate follow-up, neither in this PR:

  • The integration test's teardown leaves archived knowledge bases behind, which is how that workspace reached five figures.
  • sync-engine.ts accumulates every synced document's extracted text into one array with a page-count bound but no byte cap. That is where a bound would actually earn its keep.

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

@vercel

vercel Bot commented Aug 17, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
docs Skipped Skipped Aug 17, 2026 6:11am

Request Review

@cursor

cursor Bot commented Aug 17, 2026

Copy link
Copy Markdown

PR Summary

Medium Risk
Unpaged list endpoints can return very large result sets and heavier connector projections, trading prior hard failures for memory/latency risk on huge workspaces.

Overview
Fixes staging 500s on GET /api/knowledge when a workspace had more than 10,000 knowledge bases (often archived rows with slow cleanup). Unpaged workspace, legacy, and merged list reads no longer enforce MAX_KNOWLEDGE_BASES_PER_WORKSPACE / legacy / connector caps; paging with limit still uses limit + 1 for nextCursorKeys.

Copilot VFS rename/delete no longer loads the whole workspace with a substring search and filters in JS. They call new findActiveKnowledgeBasesByExactName (exact name, limit 2), aligned with findActiveTablesByExactName.

Removed unused constants from constants.ts. Tests cover unbounded unpaged reads, large archived sets, paging, and exact-name lookup.

Reviewed by Cursor Bugbot for commit bc7d114. Configure here.

@greptile-apps

greptile-apps Bot commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR removes fixed caps from unpaged knowledge-base and connector projections while preserving keyset limits for explicitly paged reads. It also replaces VFS name resolution through a workspace-wide search with a bounded exact-name query.

  • Makes unpaged workspace and legacy knowledge-base reads unbounded.
  • Keeps paged workspace reads at limit + 1 for cursor generation.
  • Adds an exact-name, two-row VFS lookup.
  • Updates regression and paging tests.

Confidence Score: 5/5

The PR appears safe to merge because no blocking failure remains.

No blocking failure remains.

Important Files Changed

Filename Overview
apps/sim/lib/knowledge/service.ts Removes unpaged row and connector projection caps, preserves explicit paging, and adds a bounded exact-name lookup.
apps/sim/lib/knowledge/application/knowledge-vfs.ts Switches VFS knowledge-base resolution from workspace list filtering to the exact-name service query.
apps/sim/lib/knowledge/service.test.ts Replaces cap-failure coverage with unbounded-read, cursor paging, exact-name lookup, and oversized archived-list regression coverage.
apps/sim/lib/knowledge/constants.ts Removes constants for the deleted knowledge-base and connector projection limits.

Reviews (2): Last reviewed commit: "fix(knowledge): stop capping the unpaged..." | Re-trigger Greptile

Comment thread apps/sim/lib/knowledge/service.ts Outdated
#6770 routed GET /api/knowledge through the workspace read, which carried a
10,000-row cap. Staging crossed it, and the first list request after the deploy
returned 500 with "Knowledge base list exceeds the 10000 row limit" — against
data that had served fine for days.

The cap could never have worked. Its throw was guarded by `limit === undefined`,
so it fired only for callers that had NOT asked for a page: exactly the callers
with no cursor to retry with and no way to ask for less. A paged caller never
reached it. It also read one row PAST the cap before throwing, so it refused to
serve rows it had already materialized — most of the memory was already spent.
Soft-delete cleanup reclaims archived rows only past a retention window, and not
at all where none is configured, so a workspace that archives faster than that
window crosses any fixed count on its own.

Remove it. An unpaged read is unbounded, matching the sibling internal lists
(`listTables`, workspace files), and paged callers keep their page. That fixes
the same latent 500 in the archived list, the catalog read, and the VFS name
lookup, which are all unpaged too, rather than only the surface that failed.

Two more of the same shape found while auditing for others:

- `attachConnectorTypes` threw a bare Error above its own cap, on those same
  unpaged callers. Archiving a knowledge base archives its connectors, so the
  growth curve that broke staging could not reach it — but it is the identical
  construct, and the sibling `latestJobsForTables` has no equivalent.
- The VFS path lookup read every knowledge base whose name merely CONTAINED the
  term and then exact-matched in JS, so a single-row lookup scaled with the
  workspace. It now queries the exact name and reads two rows, mirroring
  `findActiveTablesByExactName`.
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cursor review

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit bc7d114. Configure here.

@waleedlatif1
waleedlatif1 merged commit fd828f8 into staging Aug 17, 2026
31 checks passed
@waleedlatif1
waleedlatif1 deleted the fix/kb-archived-list-cap branch August 17, 2026 06:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant