Skip to content

connector: postgres sink respect column DEFAULTs for omitted columns#6699

Open
swanandx wants to merge 1 commit into
mainfrom
issue6694
Open

connector: postgres sink respect column DEFAULTs for omitted columns#6699
swanandx wants to merge 1 commit into
mainfrom
issue6694

Conversation

@swanandx

Copy link
Copy Markdown
Member

The INSERT used SELECT * from jsonb_populate_recordset, which named every column and fed NULL into columns absent from the Feldera schema, breaking DEFAULT values of columns. List only the supplied columns so Postgres applies each omitted column's DEFAULT.

Fixes #6694

Describe Manual Test Plan

relying on added tests

Checklist

  • Unit tests added/updated
  • Integration tests added/updated

Breaking Changes?

should not be.

The INSERT used `SELECT *` from jsonb_populate_recordset, which named
every column and fed NULL into columns absent from the Feldera schema,
breaking DEFAULT values of columns. List only the supplied columns so
Postgres applies each omitted column's DEFAULT.

Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>

@mythical-fred mythical-fred left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APPROVE. Neat fix for #6694: explicit column list in the jsonb_populate_recordset INSERT so Postgres applies column DEFAULTs (and NOT NULL DEFAULT columns like served_at TIMESTAMP DEFAULT now() NOT NULL) instead of SELECT * forcing NULL into every omitted column.

  • CDC path also lists the __feldera_op / __feldera_ts metadata columns explicitly — consistent with the materialized path and with the existing pipeline invariant that value_schema.fields doesn't already contain those.
  • Case-sensitive columns stay quoted (test materialized_insert_quotes_case_sensitive_columns locks that in).
  • Integration test creates a real NOT NULL DEFAULT now() column outside the Feldera schema and asserts every row's served_at was populated from the DEFAULT — the exact failure mode from #6694.

One optional follow-up (non-blocking): the CDC branch pushes cdc_op_column/cdc_ts_column into insert_columns while the materialized branch does not — if a future refactor merges those config fields into value_schema, the CDC branch will silently double-list them. A one-line debug_assert that insert_columns has no duplicates would catch that regression cheaply, but it's not worth blocking on now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A question about "Columns in the PostgreSQL table that are nullable or have default values may be omitted from the Feldera view"

3 participants