Skip to content

[SQL] Add the value of the property 'skip_unused_columns' for the table to the generated Rust code #5958

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ryzhyk merged 3 commits into from
Apr 2, 2026
Merged

[SQL] Add the value of the property 'skip_unused_columns' for the table to the generated Rust code #5958

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
116f32e
[SQL] Add the value of the property 'skip_unused_columns' for the tab…
mihaibudiu Mar 31, 2026
be7af0c
[qa] Test for skip_unused_columns.
ryzhyk Apr 2, 2026
b6854f8
[ci] apply automatic fixes
feldera-bot Apr 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
[qa] Test for skip_unused_columns. 
We have test coverage for connector-level skip_unused_columns. This commit adds
a test that checks that this attribute can also be picked up from a table-level
property. It uses the TPCH test, adding the attribute to one of the columns and
checking that the column ends up with NULL values in that column.

Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
  • Loading branch information
@ryzhyk
ryzhyk committed Apr 2, 2026
commit be7af0c5a0c9eba6f0eb302dbfbfadb7153fe213
9 changes: 6 additions & 3 deletions python/feldera/testutils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -347,8 +347,8 @@ def number_of_processed_records(pipeline: Pipeline) -> int:


def run_workload(
pipeline_name: str, tables: dict, views: List[ViewSpec], transaction: bool = True
):
pipeline_name: str, tables: dict, views: List[ViewSpec], transaction: bool = True, stop: bool = True
) -> Pipeline:
"""
Helper to run a pipeline to completion and validate the views afterwards using ad-hoc queries.

Expand Down Expand Up @@ -391,4 +391,7 @@ def run_workload(

validate_outputs(pipeline, tables, views)

pipeline.stop(force=True)
if stop:
pipeline.stop(force=True)

return pipeline
17 changes: 13 additions & 4 deletions python/tests/workloads/test_tpch.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -359,6 +359,7 @@ def tpch_tables(config: TPCHTestConfig):
R_NAME CHAR(25) NOT NULL,
R_COMMENT VARCHAR(152)
) WITH (
'skip_unused_columns' = 'true',
'materialized' = 'true',
'connectors' = '[{input_connector(config, "region")}]'
);
Expand Down Expand Up @@ -1427,13 +1428,21 @@ def tpch_test(config: TPCHTestConfig):
pipeline.modify(sql=sql)

elif config.mode == "transaction":
run_workload(
unique_pipeline_name("tpc-h-transaction"), tables, views, transaction=True
pipeline = run_workload(
unique_pipeline_name("tpc-h-transaction"), tables, views, transaction=True, stop=False
)
count = next(pipeline.query("SELECT COUNT(*) as cnt FROM REGION WHERE R_COMMENT IS NOT NULL"))["cnt"]
if count != 0:
raise RuntimeError(f"REGION.R_COMMENT is not NULL despite 'skip_unused_columns' = 'true': {count}")
pipeline.stop(force=True)
elif config.mode == "stream":
run_workload(
unique_pipeline_name("tpc-h-stream"), tables, views, transaction=False
pipeline = run_workload(
unique_pipeline_name("tpc-h-stream"), tables, views, transaction=False, stop=False
)
count = next(pipeline.query("SELECT COUNT(*) as cnt FROM REGION WHERE R_COMMENT IS NOT NULL"))["cnt"]
if count != 0:
raise RuntimeError(f"REGION.R_COMMENT is not NULL despite 'skip_unused_columns' = 'true': {count}")
pipeline.stop(force=True)
else:
raise RuntimeError(f"Unknown mode: {config.mode}")

Expand Down
Loading