[SQL] Support for the MAP_CONCAT SQL function#6515
Conversation
There was a problem hiding this comment.
Remove MAP_CONCAT from the table of unsupported map operators. Otherwise LGTM!
mythical-fred
left a comment
There was a problem hiding this comment.
Tight, well-scoped PR. New MAP_CONCAT(map1, map2) returns the right-biased merge, propagates NULL on either argument, and is correctly tagged as a Func in CalciteFunctions.java. Docs row in map.md lands in the alphabetical position; entry added to function-index.md; MAP_CONCAT removed from unsupported-operations.md. The new MapValuesFunction registration (with the Cannot be folded by Calcite comment) is the right fix for the deterministic-order issue you noted.
The unrelated MapTests migration to SqlIoTest + qst golden-table tests is also a real improvement — it removes a lot of hand-built DBSPZSetExpression boilerplate and the new TableParser.parseValue MAP branch is what makes that possible.
A few non-blocking observations:
crates/sqllib/src/map.rs:map_concat__builds a freshBTreeMapand clones every entry from both sides. TheMapalias isArc<BTreeMap<...>>, so whenrightcovers all ofleft's keys (the common-case "override" usage) you're paying forleft.len()doomed clones.right.iter().chain(left.iter().filter(|(k, _)| !right.contains_key(k)))would skip the wasted clones. Not worth blocking on — call-site sizes are usually small — but worth a follow-up ifMAP_CONCATshows up in a hot path.Map<I, T>isArc<BTreeMap>; you could also clone-from-rightandextendwithleft's missing keys to skip a fresh allocation whenrightis the only owner. Same caveat.TableParser.parseValue MAP:trimmed.split(",")andpart.split(":")are explicitly called fragile in the comment, which is fine for test fixtures, but any future value containing a literal,or:(a stringly-encoded timestamp, a nested map, a varchar with punctuation) will silently misparse rather than fail loudly. AUtilities.enforce(parts.length > 0)plus aif (trimmed.contains("{") || trimmed.contains("["))reject would at least surface the unsupported cases instead of producing wrong values.testMapConcatcovers the basic shape, NULL-propagation, key collision (right wins), and the cast-coercion case (MAP[1,2]+MAP[1.0, NULL]→{1.0: NULL}). Nice coverage; the cast-coercion case is the subtle one and you got it right.
Note: I see the linked Calcite JIRA (CALCITE-7615) is still open for the deterministic-order fix on the encoder side. Worth tracking so this doesn't ship a half-done MAP story.
Special PR number? Not particularly — but the prior MAP_VALUES "Cannot be folded by Calcite" workaround is the kind of breadcrumb that future-Mihai will thank present-Mihai for. Good comment.
Signed-off-by: Mihai Budiu <mbudiu@feldera.com>
mythical-fred
left a comment
There was a problem hiding this comment.
The MAP_CONCAT implementation itself looks right.
map_concat__clones left, extends with right —BTreeMap::extendoverwrites on duplicate key, so "keys in the second map win" matches the docs.- The n-ary fold in
ExpressionCompiler.MAP_CONCATcasts every arg to the common result type before folding pairwise; last-wins semantics carries through cleanly. Single-arg short-circuits to the arg, zero-arg raises a typed compilation error. Good shape. map_concat__is registered viasome_function2!so nullability propagation rides on the existing machinery — confirmed by the explicitMAP_CONCAT(NULL, MAP[1,2])test.- Test coverage in
MapTests.testMapConcatcovers the cases I'd ask for: happy path, NULL propagation, key collision (last-wins), heterogeneous-type promotion (INT → DECIMAL key), 1-arg, 3-arg fold. Plus the newTableParserMAP-literal parsing intools/TableParser.javato round-trip these test outputs. CustomFunctionsmovingMAP_KEYS/MAP_VALUESfromCalciteFunctionstoCustomFunctions(with the "Cannot be folded by Calcite since the result order is not deterministic" comment) is the right reason — Calcite's constant-folding would observe a different key order than ours.
Two non-blocking nits:
map_concat__doesout.extend(left.iter().map(|(k, v)| (k.clone(), v.clone()))); for aBTreeMapyou couldout.extend(left.iter().map(|(k,v)| (k.clone(), v.clone())))→ fine, butclone_fromorout = (*left).clone(); out.extend(right.iter()...)saves one full traversal on the common 2-arg path. Micro-optimization, ignore if the function only runs at the row level over small maps.- The four-row docs table cell for
MAP_CONCATis long; a<br/>between the description and the NULL-propagation clause would render more readably on narrow viewports. Cosmetic.
LGTM.
Part of #1917
Will need to be improved by solving https://issues.apache.org/jira/browse/CALCITE-7615
Describe Manual Test Plan
Ran all the new Java tests
Checklist