Skip to content

fix(microsoft-dataverse): align integration with live API docs, add table metadata tool#5481

Merged
waleedlatif1 merged 5 commits into
stagingfrom
validate/dataverse-integration
Jul 7, 2026
Merged

fix(microsoft-dataverse): align integration with live API docs, add table metadata tool#5481
waleedlatif1 merged 5 commits into
stagingfrom
validate/dataverse-integration

Conversation

@waleedlatif1

Copy link
Copy Markdown
Collaborator

Summary

  • trim environment URL/entity/record IDs across all Dataverse tools via a shared getDataverseBaseUrl util (protects against copy-paste whitespace 404s)
  • encodeURIComponent OData $select/$filter/$orderby/$expand values in list_records/get_record so filter values with &, #, quotes, etc. don't break the query string
  • fix $top being silently ignored by Dataverse when Prefer: odata.maxpagesize is also sent — only send the maxpagesize preference when the caller hasn't set an explicit $top
  • add microsoft_dataverse_get_entity_metadata tool to look up a table's entity set name, primary key/name columns, and column definitions — closes the gap where every write/query tool requires knowing the entity set name and column logical names up front. Fully covered by the existing user_impersonation scope, no new scopes added

Type of Change

  • Bug fix
  • New feature (non-breaking)

Testing

Tested manually; ran bun run lint and bun run type-check clean for the sim package.

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

…able metadata tool

- trim environment URL/entity/record IDs across all tools via shared getDataverseBaseUrl
- encodeURIComponent OData $select/$filter/$orderby/$expand values in list_records/get_record
- fix $top being silently ignored when Prefer: odata.maxpagesize is also sent
- add microsoft_dataverse_get_entity_metadata tool for table/column lookup (covered by existing user_impersonation scope, no new scopes)
@vercel

vercel Bot commented Jul 7, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
docs Skipped Skipped Jul 7, 2026 6:58pm

Request Review

@cursor

cursor Bot commented Jul 7, 2026

Copy link
Copy Markdown

PR Summary

Medium Risk
Touches many Dataverse request builders (pagination and encoding behavior changes) plus a new metadata API path; changes are mostly defensive and additive, with no new OAuth scopes.

Overview
Hardens Microsoft Dataverse Web API URL building and adds Get Table Metadata so workflows can resolve entity set names and column logical names before CRUD.

Shared URL normalization: Introduces getDataverseBaseUrl and uses it across Dataverse tools (and the upload-file API route). Environment URLs are trimmed and de-trailing-slashed; entity set names, record IDs, and related path segments are trimmed to avoid paste-whitespace 404s.

Query / paging fixes: list_records and get_record now encodeURIComponent OData $select, $filter, $orderby, and $expand values. list_records only sends Prefer: odata.maxpagesize=100 when the caller did not set $top, because Dataverse ignores $top when both are present.

New capability: microsoft_dataverse_get_entity_metadata calls EntityDefinitions(LogicalName='…') with optional $select and optional $expand on attributes. The Microsoft Dataverse block gains a Get Table Metadata operation (table logical name, optional column definitions, advanced metadata property select), registry wiring, outputs, and a discover-table-schema skill.

Reviewed by Cursor Bugbot for commit efdd643. Configure here.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit efdd643. Configure here.

Comment thread apps/sim/blocks/blocks/microsoft_dataverse.ts
@greptile-apps

greptile-apps Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR consolidates Microsoft Dataverse URL normalization into a shared getDataverseBaseUrl util, applies encodeURIComponent to OData query-option values in list_records/get_record, fixes $top being silently ignored when Prefer: odata.maxpagesize is also present, and adds the new microsoft_dataverse_get_entity_metadata tool for table schema discovery.

  • URL normalization (utils.ts): A single getDataverseBaseUrl replaces the per-file replace(/\/$/, '') pattern across all 13 tool files; each call site also .trim()s entity set names and record IDs.
  • OData encoding (list_records.ts, get_record.ts): $filter, $select, $orderby, and $expand values are now encodeURIComponent-escaped before being appended to the query string, protecting against &, #, and quote characters breaking the URL.
  • $top/odata.maxpagesize conflict (list_records.ts): The Prefer: odata.maxpagesize token is now only added when the caller has not set an explicit $top, aligning with Dataverse's documented behavior of silently ignoring $top when that preference is present.
  • New metadata tool (get_entity_metadata.ts): Fetches table entity-set name, primary key/name column, and optionally column definitions; properly doubles embedded single quotes in the OData key predicate and uses $expand=Attributes(...) nested options syntax.

Confidence Score: 5/5

Safe to merge — the changes are narrowly scoped bug fixes and a well-contained new tool with no new auth scopes.

All three categories of change (URL normalization, OData encoding, $top/$maxpagesize conflict resolution) are straightforward and their correctness can be verified by reading each call site. The new get_entity_metadata tool correctly handles the OData single-quote escaping corner case, removes the silent-catch on the success-path JSON parse, and stays within the existing OAuth scope.

No files require special attention.

Important Files Changed

Filename Overview
apps/sim/tools/microsoft_dataverse/utils.ts New shared utility: trims whitespace and strips trailing slash from environment URLs. Simple and correct.
apps/sim/tools/microsoft_dataverse/get_entity_metadata.ts New tool for table/column metadata lookup. Single-quote doubling in OData key predicate is correct; nested $expand option is well-formed; success-path JSON parse no longer has a silent catch.
apps/sim/tools/microsoft_dataverse/list_records.ts Fixes $top/$maxpagesize conflict and adds encodeURIComponent to all OData query-option values; Prefer header composition is correct.
apps/sim/tools/microsoft_dataverse/execute_function.ts Splits user-provided parameter string on the first '?' to separate inline params from alias bindings; updated description guides users on reserved-character handling.
apps/sim/tools/microsoft_dataverse/download_file.ts Adds a file output object for execution storage, resolves size from header or buffer, and falls back to 'download' for missing x-ms-file-name.
apps/sim/app/api/tools/microsoft-dataverse/upload-file/route.ts Adds a 128 MB pre-flight size guard and switches to getDataverseBaseUrl; guard fires before the network call, preventing oversized requests from reaching Dataverse.
apps/sim/blocks/blocks/microsoft_dataverse.ts Wires get_entity_metadata into the block UI and exposes facets/skip/count block fields; follows established patterns.
apps/sim/tools/microsoft_dataverse/types.ts Adds DataverseGetEntityMetadataParams/Response types, extends DataverseResponse union, and adds file/fileColumn fields to DataverseDownloadFileResponse.
apps/sim/tools/registry.ts Registers the new dataverseGetEntityMetadataTool under the expected key; import and registration are consistent.

Reviews (2): Last reviewed commit: "fix(microsoft-dataverse): fix function a..." | Re-trigger Greptile

Comment thread apps/sim/tools/microsoft_dataverse/get_entity_metadata.ts
Comment thread apps/sim/tools/microsoft_dataverse/get_entity_metadata.ts Outdated
…guard, restore file output back-compat

- execute_function: support @p1/@p2 parameter alias query-string bindings for values with reserved characters, per Dataverse Web API function docs
- upload-file route: reject files over Dataverse's 128MB single-request upload ceiling with a clear error instead of an opaque API failure
- download_file: add canonical `file` output (type: 'file') so downloaded bytes get persisted through Sim's execution file storage like every other file-download tool, while keeping the pre-existing fileContent/fileName/fileSize/mimeType fields for backwards compatibility with existing workflows
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

Ran a final comprehensive validation pass with 5 independent parallel agents covering: core CRUD tools, advanced/action tools (associate/disassociate/execute_action/execute_function/fetchxml/bulk/metadata), file-handling conventions vs sibling integrations, backwards-compatibility diffing against origin/staging, and a fresh-eyes full block/registry re-audit.

Found and fixed 3 more real issues in b04346257a:

  • execute_function: the tool's own docs told users to use @p1/@p2 parameter aliases for complex values, but the URL builder had no way to append the required ?@p1=...&@p2=... binding — fixed to split on ? and append it correctly.
  • Upload route: added a clear 400 for files over Dataverse's 128MB single-request upload ceiling instead of letting it fail opaquely against the API.
  • download_file: an agent had swapped the flat fileContent/fileName/fileSize/mimeType outputs for a canonical file: {type:'file'} output (correct for Sim's execution-file-storage auto-persistence) — but since download_file already shipped on staging with the flat fields, that would've broken existing workflows referencing them. Restored the flat fields alongside the new file field so both old and new consumers work.

Also confirmed: zero new OAuth scopes anywhere in this branch, zero backwards-incompatible removals (diffed every hunk against origin/staging), added missing count/facets/skip subBlocks that were in tool params/outputs but unreachable from the UI, and verified the discover-table-schema BlockMeta skill against real Microsoft docs.

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile please review the latest commit

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cursor please review the latest commit

@cursor

cursor Bot commented Jul 7, 2026

Copy link
Copy Markdown

Based on my review of commit b04346257a, here's my comprehensive assessment:

Strengths

1. execute_function: Parameter Alias Support

The fix correctly handles OData function parameter aliases (@p1, @p2) for values with reserved characters. The implementation:

  • Properly splits on ? to separate inline params from alias bindings
  • Appends the query suffix correctly to all three URL variants (bound to record, bound to entity set, unbound)
  • Updated the param description with clear usage instructions

Minor suggestion: Consider adding a validation or example to the description showing the exact format users should follow.

2. Upload Route: Size Guard

The 128MB limit check is well-implemented:

  • Uses a clear constant name
  • Provides helpful error message with actual file size
  • Catches the issue early before hitting the API

3. download_file: Backwards Compatibility

Excellent approach maintaining both old and new outputs:

  • Adds canonical file output for consistency with other file-download tools
  • Preserves existing fileContent, fileName, fileSize, mimeType fields
  • Adds fileColumn to outputs (was missing before)

4. Block UI: Missing Fields

Good catch adding the missing count, facets, and skip subBlocks that were in tool params but not exposed.

⚠️ Issues & Recommendations

1. execute_function: Edge Case in Parameter Parsing

const separatorIndex = rawParams.indexOf('?')
const inlineParams = separatorIndex === -1 ? rawParams : rawParams.slice(0, separatorIndex)
const aliasQuery = separatorIndex === -1 ? '' : rawParams.slice(separatorIndex + 1)

Issue: If a user accidentally includes ? in their inline parameter value without intending to use aliases, it will be misinterpreted as a separator. While this is unlikely with proper OData quoting, you might want to document this limitation.

Recommendation: Add a note in the param description warning that ? is reserved as the alias separator.

2. download_file: Inconsistent Fallback Values

const fileName = response.headers.get('x-ms-file-name') || 'download'
const fileSize = response.headers.get('x-ms-file-size') ?? ''
const mimeType = response.headers.get('mimetype') ?? 
  response.headers.get('content-type') ?? 
  'application/octet-stream'

Issue: Mixing || and ?? operators:

  • fileName uses || (treats empty string as falsy → 'download')
  • fileSize uses ?? (only treats null/undefined as nullish → keeps empty string)

Recommendation: Use consistent nullish coalescing. For fileName, consider:

const fileName = response.headers.get('x-ms-file-name') || 'downloaded-file'

3. Types: Buffer | string Union

data: Buffer | string // Buffer for direct use, string for base64-encoded data

Issue: The actual implementation always uses string (base64), but the type suggests Buffer is possible. This creates confusion.

Recommendation: Either:

  • Change type to just string since it's always base64
  • Or add a discriminated union if you plan to support both formats

4. Block Outputs: File Column Description

fileColumn: { type: 'string', description: 'File column name' },

Recommendation: Make description more specific:

fileColumn: { type: 'string', description: 'Logical name of the file column the file was uploaded to or downloaded from' },

5. Missing Test Updates

The commit adds new parameters and changes behavior but doesn't show any test updates. Consider adding tests for:

  • execute_function with alias parameters
  • upload-file size validation
  • download_file with the new file output structure

📝 Overall Assessment

Status: ✅ Approved with minor recommendations

The commit successfully addresses three distinct issues with clean, focused changes:

  1. Fixes a real limitation in execute_function where reserved characters couldn't be passed
  2. Adds proper validation for file upload size limits
  3. Modernizes download_file output structure while maintaining backwards compatibility

The code follows repository conventions and maintains consistency with existing patterns. The issues I identified are minor and mostly around edge cases, documentation clarity, and type precision.

Recommendation: This is ready to merge as-is. The minor suggestions above can be addressed in a follow-up if desired, but they're not blocking.

Open in Web Open in Cursor 

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

Thanks for the review — addressed the actionable items in `ae0cb90d3b`:

  • fileColumn description: clarified to "Logical name of the file column the file was uploaded to or downloaded from" as suggested.
  • `?` reserved separator: already documented — the `parameters` field description explicitly lists `?` in the reserved-characters set that requires alias syntax, so this is covered.
  • `Buffer | string` type: intentionally left as-is — this is the established shared convention for file-output types across the codebase (onedrive, sharepoint, and slack tools all use the identical `data: Buffer | string` shape with the same comment). Changing it only here would create an inconsistency with sibling integrations rather than fix one.
  • `||` vs `??` on fileName/fileSize/mimeType: not a bug — `fileSize` is immediately re-checked with a truthy guard (`fileSize ? parseInt(...) : ...`) right after, so `??` vs `||` is behaviorally identical there. `fileName`'s `||` is intentional (falls back to a default name when the header is an empty string, not just missing). Leaving as-is.
  • Test coverage: no existing tool handler tests exist anywhere in this integration (consistent with sibling integrations added recently), so no new tests added, matching repo convention.

Appreciate the thorough pass — let me know if any of the above reasoning doesn't hold up.

@waleedlatif1 waleedlatif1 merged commit d32b966 into staging Jul 7, 2026
16 checks passed
@waleedlatif1 waleedlatif1 deleted the validate/dataverse-integration branch July 7, 2026 19:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant