Skip to content

[SQL] Support for the MAP_CONCAT SQL function#6515

Merged
mihaibudiu merged 1 commit into
feldera:mainfrom
mihaibudiu:map_concat
Jun 24, 2026
Merged

[SQL] Support for the MAP_CONCAT SQL function#6515
mihaibudiu merged 1 commit into
feldera:mainfrom
mihaibudiu:map_concat

Conversation

@mihaibudiu

Copy link
Copy Markdown
Contributor

Part of #1917

Will need to be improved by solving https://issues.apache.org/jira/browse/CALCITE-7615

Describe Manual Test Plan

Ran all the new Java tests

Checklist

  • Unit tests added/updated
  • Integration tests added/updated
  • Documentation updated

@anandbraman anandbraman left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove MAP_CONCAT from the table of unsupported map operators. Otherwise LGTM!

@mihaibudiu
mihaibudiu added this pull request to the merge queue Jun 23, 2026
@mihaibudiu
mihaibudiu removed this pull request from the merge queue due to a manual request Jun 23, 2026
@mihaibudiu
mihaibudiu added this pull request to the merge queue Jun 23, 2026

@mythical-fred mythical-fred left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tight, well-scoped PR. New MAP_CONCAT(map1, map2) returns the right-biased merge, propagates NULL on either argument, and is correctly tagged as a Func in CalciteFunctions.java. Docs row in map.md lands in the alphabetical position; entry added to function-index.md; MAP_CONCAT removed from unsupported-operations.md. The new MapValuesFunction registration (with the Cannot be folded by Calcite comment) is the right fix for the deterministic-order issue you noted.

The unrelated MapTests migration to SqlIoTest + qst golden-table tests is also a real improvement — it removes a lot of hand-built DBSPZSetExpression boilerplate and the new TableParser.parseValue MAP branch is what makes that possible.

A few non-blocking observations:

  • crates/sqllib/src/map.rs: map_concat__ builds a fresh BTreeMap and clones every entry from both sides. The Map alias is Arc<BTreeMap<...>>, so when right covers all of left's keys (the common-case "override" usage) you're paying for left.len() doomed clones. right.iter().chain(left.iter().filter(|(k, _)| !right.contains_key(k))) would skip the wasted clones. Not worth blocking on — call-site sizes are usually small — but worth a follow-up if MAP_CONCAT shows up in a hot path.
  • Map<I, T> is Arc<BTreeMap>; you could also clone-from-right and extend with left's missing keys to skip a fresh allocation when right is the only owner. Same caveat.
  • TableParser.parseValue MAP: trimmed.split(",") and part.split(":") are explicitly called fragile in the comment, which is fine for test fixtures, but any future value containing a literal , or : (a stringly-encoded timestamp, a nested map, a varchar with punctuation) will silently misparse rather than fail loudly. A Utilities.enforce(parts.length > 0) plus a if (trimmed.contains("{") || trimmed.contains("[")) reject would at least surface the unsupported cases instead of producing wrong values.
  • testMapConcat covers the basic shape, NULL-propagation, key collision (right wins), and the cast-coercion case (MAP[1,2] + MAP[1.0, NULL]{1.0: NULL}). Nice coverage; the cast-coercion case is the subtle one and you got it right.

Note: I see the linked Calcite JIRA (CALCITE-7615) is still open for the deterministic-order fix on the encoder side. Worth tracking so this doesn't ship a half-done MAP story.

Special PR number? Not particularly — but the prior MAP_VALUES "Cannot be folded by Calcite" workaround is the kind of breadcrumb that future-Mihai will thank present-Mihai for. Good comment.

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jun 23, 2026
Signed-off-by: Mihai Budiu <mbudiu@feldera.com>
@mihaibudiu
mihaibudiu enabled auto-merge June 24, 2026 17:55
@mihaibudiu
mihaibudiu added this pull request to the merge queue Jun 24, 2026

@mythical-fred mythical-fred left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The MAP_CONCAT implementation itself looks right.

  • map_concat__ clones left, extends with right — BTreeMap::extend overwrites on duplicate key, so "keys in the second map win" matches the docs.
  • The n-ary fold in ExpressionCompiler.MAP_CONCAT casts every arg to the common result type before folding pairwise; last-wins semantics carries through cleanly. Single-arg short-circuits to the arg, zero-arg raises a typed compilation error. Good shape.
  • map_concat__ is registered via some_function2! so nullability propagation rides on the existing machinery — confirmed by the explicit MAP_CONCAT(NULL, MAP[1,2]) test.
  • Test coverage in MapTests.testMapConcat covers the cases I'd ask for: happy path, NULL propagation, key collision (last-wins), heterogeneous-type promotion (INT → DECIMAL key), 1-arg, 3-arg fold. Plus the new TableParser MAP-literal parsing in tools/TableParser.java to round-trip these test outputs.
  • CustomFunctions moving MAP_KEYS/MAP_VALUES from CalciteFunctions to CustomFunctions (with the "Cannot be folded by Calcite since the result order is not deterministic" comment) is the right reason — Calcite's constant-folding would observe a different key order than ours.

Two non-blocking nits:

  1. map_concat__ does out.extend(left.iter().map(|(k, v)| (k.clone(), v.clone()))); for a BTreeMap you could out.extend(left.iter().map(|(k,v)| (k.clone(), v.clone()))) → fine, but clone_from or out = (*left).clone(); out.extend(right.iter()...) saves one full traversal on the common 2-arg path. Micro-optimization, ignore if the function only runs at the row level over small maps.
  2. The four-row docs table cell for MAP_CONCAT is long; a <br/> between the description and the NULL-propagation clause would render more readably on narrow viewports. Cosmetic.

LGTM.

Merged via the queue into feldera:main with commit 431ca4e Jun 24, 2026
1 check passed
@mihaibudiu
mihaibudiu deleted the map_concat branch June 24, 2026 23:52
@mihaibudiu mihaibudiu mentioned this pull request Jun 25, 2026
10 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants