Skip to content

feat: add SQL transpilation for pre-aggregate sql_filter expressions#21734

Draft
IrakliJani wants to merge 1 commit intomainfrom
04-03-feat_add_sql_transpilation_for_pre-aggregate_sql_filter_expressions
Draft

feat: add SQL transpilation for pre-aggregate sql_filter expressions#21734
IrakliJani wants to merge 1 commit intomainfrom
04-03-feat_add_sql_transpilation_for_pre-aggregate_sql_filter_expressions

Conversation

@IrakliJani
Copy link
Copy Markdown
Contributor

Description:

This PR adds SQL transpilation support for pre-aggregates to handle cross-database compatibility when the source warehouse uses a different SQL dialect than DuckDB.

Key Changes:

  • Added @polyglot-sql/sdk dependency for SQL dialect transpilation between different database systems (Snowflake, BigQuery, Postgres, etc. → DuckDB)

  • Implemented transpileExploreSqlFilters function that:

    • Transpiles sqlWhere expressions from source warehouse dialects to DuckDB syntax
    • Renames column references to flattened pre-aggregate column names (e.g., statusorders_status)
    • Preserves Lightdash user attribute placeholders like ${lightdash.attributes.X} and ${ld.attr.X}
    • Handles complex SQL expressions including date functions, casts, and array operations
  • Enhanced pre-aggregate matching to allow matches when base tables have sqlWhere filters, since they can now be properly transpiled at query time

  • Updated demo configuration with a comprehensive sql_filter example showing attribute-based filtering with comma-separated values and PostgreSQL-specific syntax

  • Added comprehensive test coverage for various SQL transpilation scenarios including cross-table joins, complex date filters, and edge cases

The transpilation ensures that pre-aggregates work seamlessly across different warehouse types while maintaining the same filtering logic and user attribute support.

Copy link
Copy Markdown
Contributor Author

This stack of pull requests is managed by Graphite. Learn more about stacking.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

revert this file :) or just use it for testing

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Apr 3, 2026

🧪 Test Selection

✅ Tests that will run

Test Description
Preview Environment Deploys a preview environment for testing
Backend API Tests Runs Vitest API tests

⏭️ Tests skipped (no relevant file changes detected)

Test How to trigger manually
Frontend E2E Tests Add test-frontend to PR description
Timezone Tests Add test-timezone to PR description
CLI Tests Add test-cli to PR description

Tip: Add test-all to your PR description to run all tests.

@socket-security
Copy link
Copy Markdown

@socket-security
Copy link
Copy Markdown

Warning

Review the following alerts detected in dependencies.

According to your organization's Security Policy, it is recommended to resolve "Warn" alerts. Learn more about Socket for GitHub.

Action Severity Alert  (click "▶" to expand/collapse)
Warn High
Obfuscated code: npm entities is 91.0% likely obfuscated

Confidence: 0.91

Location: Package overview

From: pnpm-lock.yamlnpm/entities@4.5.0

ℹ Read more on: This package | This alert | What is obfuscated code?

Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev.

Suggestion: Packages should not obfuscate their code. Consider not using packages with obfuscated code.

Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore npm/entities@4.5.0. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

View full report

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Apr 3, 2026

Your preview environment pr-21734 has been deployed.

Preview environment endpoints are available at:

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Apr 3, 2026

Preview Environment

🌐 URL: https://lightdash-preview-pr-21734.lightdash.okteto.dev

📋 Logs: View in GCP Console

🔧 SSH: ./scripts/okteto-ssh.sh 21734

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants