Skip to content

fix(tables): sniff CSV delimiter and capture the real header row on import#5888

Open
waleedlatif1 wants to merge 7 commits into
stagingfrom
worktree-csv-import-sparse
Open

fix(tables): sniff CSV delimiter and capture the real header row on import#5888
waleedlatif1 wants to merge 7 commits into
stagingfrom
worktree-csv-import-sparse

Conversation

@waleedlatif1

Copy link
Copy Markdown
Collaborator

Summary

  • CSV table import derived the delimiter from the file extension only, so semicolon-delimited exports (European-locale Excel) parsed as a single column — the reported "alarms-export" bug. Now the delimiter is sniffed from the file head (, ; \t |), quote-aware, with the extension only as a fallback.
  • Header derivation used Object.keys(rows[0]); with relax_column_count a ragged/sparse first data row dropped its trailing columns from schema inference and mapping. The true header row is now captured via the csv-parse columns callback on every path — fixes the "sparse csv didn't work" report.
  • New detectCsvDelimiter (trial-parse each candidate, pick most columns, tie-break on row-width consistency) mirrors PapaParse/csv.Sniffer, and sniffCsvDelimiterFromStream (64KB peek-then-replay) keeps the streaming paths memory-bounded on multi-GB files.
  • Wired into all consumers: create route, append/replace route, background runner, client dialog preview, and parseFileRows (copilot).

Type of Change

  • Bug fix

Testing

  • 442 table + route tests pass (49 new/updated covering delimiter detection, ragged headers, stream replay, error/destroy/empty edge cases)
  • Verified against the real 61 MB alarms-export file: comma → 1 column, semicolon → 34 columns / 1249 records; sniffer detects ;
  • Typecheck clean, biome clean, check:api-validation passes

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

@vercel

vercel Bot commented Jul 23, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
docs Skipped Skipped Jul 23, 2026 5:58pm

Request Review

@cursor

cursor Bot commented Jul 23, 2026

Copy link
Copy Markdown

PR Summary

Medium Risk
Touches core table import parsing across HTTP, background jobs, and UI preview; wrong delimiter/header logic could mis-map columns or corrupt imports, but behavior is heavily tested and scoped to CSV parsing.

Overview
CSV imports no longer assume comma/tab from the file extension. The first 64KB is trial-parsed to pick ,, ;, \t, or | (quote-aware scoring); extension/TSV only supplies a fallback. Streaming uploads use sniffCsvDelimiterFromStream to peek then replay the file without buffering multi-GB imports.

Header/schema inference no longer uses Object.keys(rows[0]). createCsvParser / parseCsvBuffer capture the real header row via a csv-parse columns callback (with duplicate-name deduping), including ragged first data rows—used on sync routes, the background import runner, the import dialog preview (detectCsvDelimiter aligned with the server), and parseFileRows for copilot.

Reviewed by Cursor Bugbot for commit 35156cc. Configure here.

@greptile-apps

greptile-apps Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

Improves CSV imports by detecting delimiters from bounded file prefixes and preserving the parser’s complete header row.

  • Adds quote-aware delimiter scoring with extension-based fallback.
  • Replays sniffed stream chunks into create, append, replace, and background import paths.
  • Uses captured headers for schema inference and mapping when initial data rows are sparse.
  • Aligns client previews and Copilot file parsing with server-side delimiter detection.
  • Adds regression coverage for delimiter ranking, stream replay, incomplete samples, duplicate headers, and ragged rows.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failures remain.

Important Files Changed

Filename Overview
apps/sim/lib/table/import.ts Adds shared delimiter detection, header capture and deduplication, and integrates sniffing into buffered file parsing.
apps/sim/lib/table/csv-delimiter-stream.ts Adds bounded stream-prefix sniffing with byte-exact replay and source cleanup.
apps/sim/lib/table/import-runner.ts Applies detected delimiters and authoritative CSV headers to background imports.
apps/sim/app/api/table/import-csv/route.ts Uses stream-based delimiter detection and captured headers when creating tables.
apps/sim/app/api/table/[tableId]/import/route.ts Uses sniffed delimiters and complete headers for append and replace imports.
apps/sim/app/workspace/[workspaceId]/tables/components/import-csv-dialog/import-csv-dialog.tsx Aligns client-side import previews with the server’s delimiter-sniffing window and fallback behavior.
apps/sim/lib/table/import.test.ts Adds regression tests for delimiter selection, ragged headers, duplicate headers, stream replay, cleanup, and boundary conditions.

Reviews (6): Last reviewed commit: "fix(tables): observe EOF at the exact sn..." | Re-trigger Greptile

Comment thread apps/sim/lib/table/import.ts Outdated
Comment thread apps/sim/lib/table/csv-delimiter-stream.ts Outdated
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cursor review

…mport

Table CSV import derived the delimiter purely from the file extension, so
semicolon-delimited exports (European-locale Excel) parsed as a single column.
Header derivation also read Object.keys(rows[0]), so with relax_column_count a
ragged/sparse first data row dropped its trailing columns from schema inference.

- Add detectCsvDelimiter: trial-parse the head against ,/;/tab/| and pick the
  candidate with the most columns (tie-break on row-width consistency), quote-aware
  so separators inside quoted cells don't skew the guess; extension is now only the
  fallback
- Add sniffCsvDelimiterFromStream: memory-bounded (64KB) peek-then-replay for the
  streaming import paths so multi-GB files sniff without buffering
- Capture the true header row via the csv-parse columns callback on every path,
  fixing dropped columns when the first row is short
- Wire into the create route, append/replace route, background runner, client
  dialog preview, and parseFileRows (copilot)
… buffer

Addresses review findings on the delimiter sniffer:

- Rank candidates by row-width consistency first (modal column count), then column
  count. Ranking on column count alone let a comma that appears in a semicolon
  file's unquoted header win over the real separator; the wide-but-ragged split
  now loses to the uniform one.
- Move the partial-trailing-line trim into detectCsvDelimiter so every path
  (stream, client preview, parseFileRows) prepares the sniff sample identically
  and can't disagree on the delimiter.
- Cap the stream peeker's detection copy at the sniff window via Buffer.concat's
  length arg and replay the buffered chunks by reference, so an oversized source
  chunk can't break the bounded-memory contract.
@waleedlatif1
waleedlatif1 force-pushed the worktree-csv-import-sparse branch from a5d29bc to a6cc15c Compare July 23, 2026 15:50
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cursor review

Comment thread apps/sim/lib/table/import.ts Outdated
Comment thread apps/sim/lib/table/import.ts Outdated
Comment thread apps/sim/lib/table/import.ts
Addresses round-2 review findings:

- Score each delimiter candidate by modalWidth * consistency instead of
  consistency alone. Consistency-only let a separator that appears uniformly
  inside values (e.g. a pipe once per row) beat a real delimiter whose rows are
  legitimately ragged; the product rewards a split that is both wide and uniform,
  with width breaking ties. Genuinely symmetric files (two valid columns under
  either separator) fall back to the global-default candidate order.
- Dedupe the captured header row to match the parser's record keys. csv-parse
  collapses duplicate column names onto one key (last value wins), so reporting
  the raw duplicates made schema inference invent a phantom empty column and
  could fail mapping validation. dedupeHeaders keeps first-occurrence order,
  case-sensitively, matching the emitted keys.
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cursor review

Comment thread apps/sim/lib/table/import.ts Outdated
Comment thread apps/sim/lib/table/import.ts
detectCsvDelimiter always trimmed to the last newline, so a complete file with
no trailing newline lost its final data row from scoring — leaving the header
alone and letting a header-widening separator beat the real one. It now takes a
`complete` flag: the trim runs only for truncated prefixes, where a mid-record
cut must be dropped. The stream sniffer passes it from `exhausted`, and the
buffered callers (client preview, parseFileRows) pass it when the sample covers
the whole file.
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cursor review

Comment thread apps/sim/lib/table/csv-delimiter-stream.ts Outdated
Hoist the csv-parse options out of the multi-line parse() call so the
`// double-cast-allowed` annotation sits directly above the `as unknown as`
token — the strict boundary audit only matches an annotation within three lines
of the cast. Also simplify the stream sniffer's completeness check to `exhausted`
(the loop only ends early on end-of-stream, so it already means the whole file
fit the sniff window).
The stream sniffer's read loop stopped as soon as the buffered size reached the
sniff window, so a file whose size is exactly the window never triggered the
extra read that observes end-of-stream — it was judged a truncated prefix and
dropped its final newline-less row, disagreeing with the buffered callers (which
mark that size complete). Read while size <= window so the boundary case sees EOF
and `exhausted` (hence `complete`) is set correctly. Regression test added.
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cursor review

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cursor review

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 35156cc. Configure here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant