Skip to content

[FEATURE] OData push-down: $select strips columns used only in WHERE/ORDER BY, so the client-side filter silently drops all rows #682

Closed

Description

@jeffreyaven

Summary

When a column appears in WHERE (or ORDER BY) but not in the SELECT list, the
OData query-option push-down emits $select with only the projected columns and
drops the predicate column from the upstream response. Because push-down keeps the
client-side WHERE authoritative and re-applies it locally, the predicate column is
now missing from every row, the filter matches nothing, and the query returns zero
rows despite an HTTP 200 carrying matching data
. It fails silently.

Repro

Provider: entra_id (Microsoft Graph), run with --http.log.enabled.

SELECT id, displayName FROM entra_id.users.users
WHERE userType = 'Member';

Outgoing request — $select has no userType:

http request url: 'https://graph.microsoft.com/v1.0/users?$filter=userType eq 'Member'&$select=id,displayName', method: 'GET'

Graph returns 200 with 3 matching rows (only id,displayName — no userType).
Final result set is empty:

|----|-------------|
| id | displayName |
|----|-------------|

The 3 rows were discarded by the client-side re-filter because userType is absent
from the response. Adding the predicate column to the projection confirms the cause
(returns the 3 rows):

SELECT id, displayName, userType FROM entra_id.users.users
WHERE userType = 'Member';   -- $select=id,displayName,userType

Root cause

internal/stackql/pushdown/pushdown.go, buildPushdownIntent: the projection placed
in the PushdownIntent is taken from the SELECT list only
(extractProjectionAndCount(sel.SelectExprs)). The WHERE/ORDER BY columns are
collected separately (for $filter/$orderby) and never folded into the projection,
so the $select any-sdk renders strips them.

The wire request itself is valid OData — Graph filters server-side and projects
correctly. The defect is that stackql re-applies WHERE/ORDER BY client-side
and the fetched rows no longer carry the needed column. That client-side dependency
is knowledge only stackql has, so the fix belongs here.

Fix

In buildPushdownIntent, set the intent projection to the union of the SELECT,
predicate, and order-by columns:

projection = SELECT-cols ∪ WHERE-cols ∪ ORDER BY-cols

This affects only the pushed $select; the SQL output projection is unchanged (still
just the SELECT list, so SELECT id, displayName ... WHERE userType = ... still
outputs two columns). Guard: no-op when the projection is empty — SELECT * pushes no
$select, so the bug cannot occur there.

Tests

Add push-down cases against the no-auth stackql_native_test.odata.people mock:

  • filter on unprojected column: SELECT name FROM ... WHERE city = 'NYC' asserts the
    wire $select includes city and the result set is non-empty.
  • order-by on unprojected column: SELECT name FROM ... ORDER BY age asserts $select
    includes age.

Note

Whichever columns get added to $select must satisfy the select supportedColumns
allowlist. Where select/filter/orderBy share one allowlist (e.g. entra_id) this
is automatic; a provider with divergent allowlists could otherwise trip the
all-or-nothing $select gate — worth covering in the fix.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions