Skip to content

Continue Avro validation after missing nullable fields#6698

Draft
xiasongh wants to merge 1 commit into
feldera:mainfrom
xiasongh:fix-avro-schema-validation
Draft

Continue Avro validation after missing nullable fields#6698
xiasongh wants to merge 1 commit into
feldera:mainfrom
xiasongh:fix-avro-schema-validation

Conversation

@xiasongh

Copy link
Copy Markdown
Contributor

Summary

  • continue validating later SQL columns when a nullable column is absent from the Avro schema
  • add a regression covering a missing nullable column followed by a missing required column

Root cause

validate_struct_schema returned success for the entire record when it encountered one missing nullable column. This skipped validation of every subsequent column, allowing missing required or incompatible columns to pass startup validation and fail later while parsing data.

Impact

Missing nullable columns remain supported for schema evolution, while all other declared SQL columns are still checked for compatibility.

Validation

  • cargo test -p dbsp_adapters format::avro:: --lib (46 passed on current upstream main)
  • cargo test -p dbsp_adapters missing_nullable_field_does_not_stop_validation --lib (passed on the published fork branch)
  • cargo fmt --all -- --check

@xiasongh xiasongh changed the title [codex] Continue Avro validation after missing nullable fields Continue Avro validation after missing nullable fields Jul 22, 2026
@gz
gz requested a review from mihaibudiu July 22, 2026 20:27
@xiasongh xiasongh closed this Jul 22, 2026
@xiasongh xiasongh reopened this Jul 22, 2026
@mihaibudiu

Copy link
Copy Markdown
Contributor

Why is this a draft? Can we merge it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants