fix(connectors): index Office documents and PDFs from SharePoint and OneDrive - #6785

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
docs	Skipped		Aug 17, 2026 10:13pm

PR Summary

Medium Risk
Changes connector sync and knowledge-base indexing for Microsoft Graph drives and centralizes file parsing; behavior is well covered by tests but affects production document content and re-index paths.

Overview
SharePoint and OneDrive no longer drop Office and PDF files during listing because of a plain-text-only extension filter. They use shared isIndexableConnectorFile and extractConnectorText so downloads stay as bytes and binary formats go through the same parseBuffer path as manual uploads, while existing text formats still decode as UTF-8 without re-parsing.

Failed or placeholder extractions (legacy OLE, image-only decks, empty PDFs) surface as skipped documents via ConnectorTextExtractionError and actionable skip reasons instead of silent drops or junk in the index. Listing logs now count unsupported extensions per folder so “success, 0 documents” is easier to distinguish from a bad path.

lib/file-parsers replaces lazy require() registration with a static Map registry (avoids a silent empty registry), adds OpenDocumentParser and more extension aliases, and marks degraded doc/ppt fallback output with metadata.degraded for automated callers.

^{Reviewed by Cursor Bugbot for commit 1bb1b2b. Configure here.}

Greptile Summary

The PR enables SharePoint and OneDrive connectors to index Office, PDF, and OpenDocument files while preserving the existing decoding path for text formats.

• Centralizes connector extension detection and binary-document text extraction.
• Routes supported binary formats through the shared parser registry and decompression safeguards.
• Marks degraded legacy parser fallbacks as failed connector documents instead of indexing diagnostic or scraped content.
• Adds parser-registry, format-routing, and connector regression coverage.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains; the previously reported legacy parser fallback issue is addressed by rejecting degraded extraction results and preserving them as visible failed documents.

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  List[SharePoint or OneDrive listing] --> Filter{Indexable extension?}
  Filter -- No --> Log[Count unsupported extension]
  Filter -- Yes --> Stub[Create deferred document stub]
  Stub --> Download[Download with byte limit]
  Download --> Kind{Text or binary?}
  Kind -- Text --> Decode[UTF-8 or HTML-to-text]
  Kind -- Binary --> Parse[Shared file parser]
  Parse --> Quality{Content usable and not degraded?}
  Quality -- Yes --> Index[Index extracted text]
  Quality -- No --> Skip[Persist visible failed/skipped document]
  Decode --> Index

_{Reviews (3): Last reviewed commit: "fix(file-parsers): resolve the parser re..." | Re-trigger Greptile}

Follow-up: verified the parsers against real fixtures, found a second defect

Question raised: do .pptx files need an additional parser? No — but characterising them turned up something that had to be fixed before this is safe to ship.

I built genuine OOXML archives with jszip and ran the real parser classes (not mocks). Results:

DocParser and PptxParser never throw — that is deliberate, so an interactive upload always shows the user something. But a connector sync would embed those placeholders and ZIP internals into the vector index at scale, and a placeholder sentence is indistinguishable from real content to the retriever.

Fix

Rather than have callers sniff extractionMethod === 'fallback', the parsers now declare it: a degraded?: boolean on FileParseMetadata, set by exactly those two fallback paths. DocParser's plaintext-fallback branch is intentionally not flagged — a text file misnamed .doc is a genuine extraction.

extractConnectorText raises ConnectorTextExtractionError when a parsed format returns degraded or blank content, and SharePoint/OneDrive map it to markSkipped(...) through the existing path oversized files already use. The file then shows up in the knowledge base as a failed row with an actionable reason ("Re-save it as PPTX to index it") instead of being silently dropped or indexed as junk.

The upload path is unaffected — it ignores the new flag.

Also verified while I was in here

• Runtime boundary. parseBuffer was not previously proven inside the Trigger.dev worker, which is where connector syncs run — the exact hazard CLAUDE.md warns about. It is proven: background/knowledge-processing.ts is a Trigger.dev task that reaches parseBuffer via processDocumentAsync, and its machine comment already reads large-1x // needed for large PDF processing.
• Machine sizing. knowledge-connector-sync runs on large-2x (4 vCPU / 8 GB, maxDuration 1800s) — strictly better provisioned than the large-1x task that already parses large PDFs. No new OOM risk.
• Parser-registry fragility (not fixed here, worth knowing). file-parsers/index.ts loads every parser with a bare require() inside a per-parser try/catch that only logs. Under ESM the whole registry silently comes back empty and parseBuffer reports Unsupported file type: docx. Supported types are: with an empty list. It works in the Next/Trigger runtimes, but a bundler change would degrade to "unsupported type" for every format rather than failing loudly.
• xlsx is very permissive — parseBuffer(Buffer.from('not a spreadsheet'), 'xlsx') "succeeds" and echoes the input as a one-cell sheet. Harmless (the content is the file's text) but noted.

Tests

New apps/sim/lib/file-parsers/degraded-extraction.test.ts pins the degraded contract to real parser behaviour using genuine OOXML fixtures, so if a parser stops setting the flag this breaks. Plus connector-level tests that a degraded file becomes a skipped document with the right reason, and that a text file still bypasses the parsers entirely.

Both new guards were confirmed able to fail: removing degraded: true from PptxParser reddens 2 fixture tests; removing the degraded check in extractConnectorText reddens the unit test and the connector-level skip test.

• vitest run connectors/ lib/file-parsers/ lib/knowledge/ — 1243 passed (76 files)
• tsgo --noEmit — clean apart from the pre-existing unrelated Cannot find module 'mssql'
• bun run check:audits — 29/29

Format audit: what was missing, and what is genuinely impossible

Registered in this push — all already parseable, just never wired up

Verified with real fixtures built via jszip/SheetJS, not assumed. This mattered: officeparser identifies a Buffer by sniffing content with file-type, ignoring the name we pass it, so routing had to be measured rather than reasoned about.

Zero new dependencies. .xlsm in particular is everywhere in enterprise SharePoint.

Why the degraded ones are degraded, and what can actually be done

.doc and .ppt (pre-2007) — these are OLE2 compound binaries, a completely different container from the post-2007 ZIP packages. mammoth and officeparser only read OOXML, so there is nothing to fall back on. This is fixable, but needs a dependency: word-extractor handles legacy .doc well. Legacy .ppt has no maintained JS library — realistically it needs LibreOffice/unoconv, i.e. a conversion service, not a parser. Note .xls is not in this bucket: SheetJS reads BIFF natively, which is why it already works. Meanwhile the skip reason now tells the user to re-save as DOCX/PPTX, which genuinely fixes it in seconds.

Image-only .pptx and scanned PDFs — there is legitimately no text in the file. The bytes are pixels. The only answer is OCR (Tesseract, or a cloud vision API), which is a different class of feature: slow, costly, and a product decision. Nothing a parser can do.

.rtf — deliberately left unsupported, and this is worth calling out because it was a latent trap: DocParser's plaintext-fallback branch accepts RTF (it is >90% printable ASCII) and returns {\\rtf1\\ansi\\deff0… as if it were prose, with no degraded flag. Had I added rtf to the accepted set, we would have silently indexed control words. Fixing it properly needs a dependency (rtf-parser); hand-rolling RTF tokenization is exactly the kind of thing I am not going to do here. It is now excluded and covered by a test.

.msg / .eml — common in document libraries. Both are solvable with dependencies (@kenjiuno/msgreader, mailparser). Worth doing if customers ask; not in scope here.

.one (OneNote), .pages/.key/.numbers, .vsdx — legitimately nothing. OneNote and the Apple iWork formats store text as proprietary binary (iWork uses Snappy-compressed protobuf in an IWA stream) with no maintained JS reader. Visio is an OOXML zip so it is theoretically reachable, but officeparser does not support it and the text is scattered across page XML — a custom extractor, low value.

Happy to add word-extractor (legacy .doc) and/or the email parsers in a follow-up if you want those — say the word and I will open a separate PR. I did not add dependencies unilaterally to a public repo.

Two defects found while auditing

1. The registry could silently support nothing. It loaded every parser with require() inside per-parser try/catch blocks that only logged. Under any loader where those calls fail, the registry came back empty and every format reported Unsupported file type: docx. Supported types are: — note the empty list, which is the tell. Every parser dependency is a regular, non-optional one, so a resolution failure should be loud. Now static imports; the heavy extraction libraries stay deferred inside the individual parsers, and connectors import the registry lazily so the ~60 connectors that never touch a file do not pull SheetJS into their graph.

2. index.test.ts mocks the module it tests. Lines 67–92 replace @/lib/file-parsers with a stub parseFile/isSupportedFileType backed by a hardcoded table, so it validates its own fake routing and the real registry had no coverage at all. That is precisely how defect 1 stayed invisible. Rather than rewrite that file, I added registry.test.ts against the real module: it gates every member of SupportedFileType on having a registered parser that supports buffer parsing, so adding a type to the union without wiring it up now fails a test. parser-formats.test.ts then proves each extension reaches a parser that actually handles that container, end-to-end through parseBuffer — a registry entry pointing at the wrong parser would pass the first test and fail the second.

Verification

• vitest run lib/ connectors/ app/api/files/ app/api/tools/file/ — 14,805 passed, 29 skipped (1,024 files). Deliberately wide because the static-import conversion touches every parser consumer.
• vitest run lib/file-parsers/ connectors/ — 829 passed
• tsgo --noEmit — clean apart from the pre-existing unrelated Cannot find module 'mssql'
• bun run check:audits — 29/29

@greptile

@cursor review

Self-review found one defect I introduced — fixed in 1bb1b2b5

Pressure-testing the "strictly better" claim rather than asserting it turned up a regression in my own registry rewrite.

Converting the registry to static imports also changed extension lookup from Object.keys(parsers).includes(ext) to a bracket read on an object literal — which resolves inherited keys. Measured, not theorised:

constructor      isSupportedFileType=false  parseBuffer="Parser for constructor does not support buffer parsing"
toString         isSupportedFileType=false  parseBuffer="Unsupported file type: tostring…"

PARSERS['constructor'] returned Object — truthy, with no parse methods — so a caller-supplied extension of constructor took the wrong branch, and parseFile would have raised a TypeError: parser.parseFile is not a function instead of a clean unsupported-type error. It also disagreed with isSupportedFileType, which used Object.hasOwn and correctly said false for the same input. The old Object.keys().includes() form was immune, so this was mine, not pre-existing.

Low severity — it needs a file literally named *.constructor — but it is exactly the kind of inconsistency that should not ship. Now a Map, which has no prototype chain, so lookup and support check agree by construction. isSupportedFileType also guards a non-string argument, which the try/catch it replaced used to absorb. Covered by it.each over constructor/toString/valueOf/hasOwnProperty/__proto__.

Also verified, since it was the open question on the static-import change

I flagged earlier that parseBuffer had to work in the Trigger.dev worker. Confirmed against production rather than inferred: PDFs uploaded today on v20260817.1 processed to completed with real chunk counts, so every parser module already resolves in that runtime and the require() registry was not silently empty in prod. This change therefore removes a latent hazard and makes it testable — it is not papering over a live outage, and it does not alter which modules the bundler includes (a require() of a static string literal is statically analysable too).

Blast radius for the connector behaviour change is small: 5 active SharePoint connectors in prod, zero OneDrive.

• vitest run lib/ connectors/ app/api/files/ app/api/tools/file/ — 14,811 passed, 29 skipped
• tsgo --noEmit — clean apart from the pre-existing unrelated mssql
• bun run check:audits — 29/29

@greptile

@cursor review