Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
75 commits
Select commit Hold shift + click to select a range
8810072
Checkpoint: Initial adapter bootstrapping and e2e tests with Docker C…
mdrakiburrahman Mar 30, 2026
04438af
Fixed catalog gen and add some steps for accessing dbt UI
mdrakiburrahman Mar 30, 2026
44448ab
Materialize Adventureworks as Delta Table and use DuckDB to assert ro…
mdrakiburrahman Mar 30, 2026
6635196
Adding a Kafka sales table and ensuring it fires IVM
mdrakiburrahman Mar 30, 2026
e5af63c
Materialize the Kafka topic as Delta table and use that instead in th…
mdrakiburrahman Mar 30, 2026
709b25a
Use sqlglot instead of brittle RegEx to parse Feldera SQL dialect
mdrakiburrahman Mar 31, 2026
04b7b38
Break apart contributing and customer facing README
mdrakiburrahman Mar 31, 2026
a06c699
Update changelog
mdrakiburrahman Mar 31, 2026
d8afa71
Merge branch 'main' into dev/mdrrahman/dbt-feldera
mdrakiburrahman Mar 31, 2026
0283a3d
Proofread
mdrakiburrahman Mar 31, 2026
34d0e35
Remove test_string_size override to implement the base behavior
mdrakiburrahman Mar 31, 2026
7ceaa8f
Differentiate between float vs numeric
mdrakiburrahman Apr 1, 2026
3fb8dcc
Update docstring to mention Feldera transactions
mdrakiburrahman Apr 1, 2026
56de322
Use NOW and add a test for it
mdrakiburrahman Apr 1, 2026
a870d0b
Update docstring for the expand column types
mdrakiburrahman Apr 1, 2026
22fd48e
Use better type inference in seed via _infer_column_types.
mdrakiburrahman Apr 1, 2026
c10e3d0
Add an idempotent seed uploader and downloader
mdrakiburrahman Apr 1, 2026
dc4df22
Remove seeds from git in origin
mdrakiburrahman Apr 1, 2026
24c7b86
Remove the models and macros as well
mdrakiburrahman Apr 1, 2026
ec8cf4c
Add a gitkeep to the seed folders so they are visible in git tree
mdrakiburrahman Apr 1, 2026
044063c
Merge branch 'main' into dev/mdrrahman/dbt-feldera
mdrakiburrahman Apr 1, 2026
3f6809b
Remove GitHub action changes - no tests in publish and remove CI test…
mdrakiburrahman Apr 1, 2026
0c70a30
Use the ci-post-release.yml sed to change the version
mdrakiburrahman Apr 1, 2026
2f2dcb9
Update changelog
mdrakiburrahman Apr 1, 2026
ec20ba2
Use Pipeline client instead of FelderaClient.get_pipeline and a few o…
mdrakiburrahman Apr 1, 2026
8046bf6
compiling_states needs to be a tuple in PipelineStateManager due to P…
mdrakiburrahman Apr 1, 2026
a07217d
Attempt to lint for CI pipeline
mdrakiburrahman Apr 1, 2026
2cafef7
Merge branch 'main' into dev/mdrrahman/dbt-feldera
mdrakiburrahman Apr 1, 2026
fd88b16
Merge branch 'main' into dev/mdrrahman/dbt-feldera
mdrakiburrahman Apr 2, 2026
8700b56
fix(py): fix ruff isort import ordering in dbt-feldera
mdrakiburrahman Apr 3, 2026
7e05d2f
Merge branch 'main' into dev/mdrrahman/dbt-feldera
mdrakiburrahman Apr 5, 2026
0280c4d
fix(py): pin ruff==0.9.10 in dbt-feldera to match CI pre-commit
mdrakiburrahman Apr 5, 2026
f698020
Update docstring to make it clear that this adapter uses continuous q…
mdrakiburrahman Apr 5, 2026
d94f0c8
Add documentation for TYPE_LABELS
mdrakiburrahman Apr 5, 2026
b5fb950
Update param docstring for auto_begin
mdrakiburrahman Apr 5, 2026
9ecf1b2
Turn positional tuples into a ColumnDescription NamedTuple
mdrakiburrahman Apr 5, 2026
8413b74
Add a docstring to connection.execute that it delegates to the cursor…
mdrakiburrahman Apr 5, 2026
018cd38
Update docstring for cursor.execute
mdrakiburrahman Apr 5, 2026
64afc65
Add a docstring to SqlIntent to make it clear which enum is supported…
mdrakiburrahman Apr 5, 2026
5c3dd4e
Add a small docstring to DATA_INGRESS
mdrakiburrahman Apr 5, 2026
de5b06f
Merge branch 'main' into dev/mdrrahman/dbt-feldera
mdrakiburrahman Apr 7, 2026
1859709
Merge branch 'main' into dev/mdrrahman/dbt-feldera
mdrakiburrahman Apr 8, 2026
4d43b45
Clean .temp dir
mdrakiburrahman Apr 8, 2026
2687e25
Merge branch 'main' into dev/mdrrahman/dbt-feldera
mdrakiburrahman Apr 11, 2026
95a54f7
Merge branch 'main' into dev/mdrrahman/dbt-feldera
mdrakiburrahman Apr 11, 2026
b859260
Merge branch 'main' into dev/mdrrahman/dbt-feldera
mdrakiburrahman Apr 12, 2026
6f68d81
Merge branch 'feldera:main' into dev/mdrrahman/dbt-feldera
mdrakiburrahman Apr 13, 2026
0a45510
Merge branch 'main' into dev/mdrrahman/dbt-feldera
mdrakiburrahman Apr 14, 2026
1d8fdcd
Move DuckDB into a container for tests so we don't need host mount
mdrakiburrahman Apr 14, 2026
749e793
Add adventureworks SQL scripts to git, remove .gitkeep files
mdrakiburrahman Apr 15, 2026
9aa8846
Pass 1 through PR review comments
mdrakiburrahman Apr 15, 2026
37f077e
Implement Generator in sqlglot to override FLOAT and map it to REAL
mdrakiburrahman Apr 15, 2026
edf302b
Merge branch 'main' into dev/mdrrahman/dbt-feldera
mdrakiburrahman Apr 15, 2026
a64ae10
Bump version to 0.288.0
mdrakiburrahman Apr 15, 2026
da6aa5e
Map FLOAT to REAL in dbt macro
mdrakiburrahman Apr 15, 2026
ca62160
Update README
mdrakiburrahman Apr 15, 2026
29c6c7c
Remove incremental materialization support, it's not required in Feldera
mdrakiburrahman Apr 15, 2026
da853cb
Changed Jinja macro to stored = true
mdrakiburrahman Apr 15, 2026
28fa16f
Remove ambiguous "registered" reference
mdrakiburrahman Apr 15, 2026
ac59e4d
Make TestFelderaColumn dynamic
mdrakiburrahman Apr 15, 2026
b5482d3
Remove external from FelderaRelationType
mdrakiburrahman Apr 15, 2026
7465e32
Remove test for no columns
mdrakiburrahman Apr 15, 2026
a6f2801
Factor out CATALOG_COLUMNS
mdrakiburrahman Apr 15, 2026
e4c50f7
Remove INTERVAL mapping in agate and run integration tests green
mdrakiburrahman Apr 15, 2026
dc3416c
Add a _wait_for_pipeline_idle to deterministically poll pipeline to p…
mdrakiburrahman Apr 15, 2026
b0150a6
Add lattice tests that are data driven
mdrakiburrahman Apr 15, 2026
3f04fb6
Rename test to test_remove_nonexistent_does_not_throw
mdrakiburrahman Apr 15, 2026
393d5cd
Crash test if Kafka topic isn't up and dump logs
mdrakiburrahman Apr 15, 2026
2d8c874
Update docstring for update_with_views
mdrakiburrahman Apr 15, 2026
ab70529
Lint
mdrakiburrahman Apr 15, 2026
61123e3
Dupe line
mdrakiburrahman Apr 16, 2026
05e7eb5
Remove NOW and use MD5 hash
mdrakiburrahman Apr 16, 2026
e4e07b6
There is no relationship between connectors and views being stored
mdrakiburrahman Apr 16, 2026
e1ce519
Add DBT_THREADS for faster integration tests
mdrakiburrahman Apr 16, 2026
b0dca82
Ruff the integration test folder
mdrakiburrahman Apr 16, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update docstring to make it clear that this adapter uses continuous q…
…ueries and not ad-hoc
  • Loading branch information
mdrakiburrahman committed Apr 5, 2026
commit f69802085d7b567605b77de471d51182be8a368c
1 change: 1 addition & 0 deletions .temp/dbt-confluent
Submodule dbt-confluent added at b3a7f4
50 changes: 38 additions & 12 deletions python/dbt-feldera/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,34 @@ the DBSP incremental computation engine. It automatically incrementalizes
_every_ SQL query without watermarks, scans, or `MERGE`. When input data
changes, only affected output rows are recomputed.

> [!IMPORTANT]
> **This adapter deploys
> [continuous pipelines](https://docs.feldera.com/pipelines), not
> [ad-hoc queries](https://docs.feldera.com/sql/ad-hoc).**
>
> Feldera supports two modes of query execution:
>
> - **Continuous pipelines** compile SQL into an incremental dataflow that runs
> indefinitely, processing every input change as it arrives in near real-time.
>
> - **Ad-hoc queries** are one-shot batch queries executed by
> [DataFusion](https://datafusion.apache.org/) against the state of
> [materialized tables and views](https://docs.feldera.com/sql/materialized).
> They exist primarily for development and debugging.
>
> When you run `dbt run`, this adapter assembles your models into a Feldera
Comment thread
mdrakiburrahman marked this conversation as resolved.
> pipeline program, compiles it, and **starts a continuously running pipeline**.
> The pipeline keeps processing input changes and updating outputs until it is
> explicitly stopped. This differs from typical batch-oriented dbt adapters where `dbt run` executes
> a query once, processes a batch of data and exits.

## Key features

- **Automatic incremental view maintenance (IVM)** — Feldera's DBSP engine
incrementalizes any SQL query out of the box. No manual merge logic or
watermark tuning required.
- **Streaming-native materializations** — first-class support for continuous
pipelines alongside standard dbt materializations.
- **Continuous pipeline deployment** — `dbt run` compiles and starts a
long-running Feldera pipeline; it does not execute one-shot queries.
- **Connector integration** — attach Kafka, Delta Lake, S3, and HTTP
connectors directly to models via configuration.
- **Easy setup** — pure Python adapter with no ODBC driver needed.
Expand Down Expand Up @@ -59,16 +80,18 @@ my_project:
### Concept mapping

Feldera uses different terminology than traditional databases. Here's how dbt
concepts map to Feldera:

| dbt concept | Feldera concept | Description |
| ----------------------------- | ----------------- | ----------------------------------------------- |
| `database` | _(unused)_ | Set to any string (e.g. `"default"`) |
| `schema` | Pipeline name | Each dbt schema maps to one Feldera pipeline |
| `table` materialization | Input table | External data source (Kafka, HTTP, S3) |
| `view` materialization | View | Intermediate SQL transform |
| `incremental` materialization | Materialized view | IVM-backed output, queryable ad-hoc |
| `seed` | Table + HTTP push | Schema registered, data pushed via HTTP ingress |
concepts map to Feldera. Every materialization contributes SQL to a
**continuously running pipeline** — nothing is executed as a one-shot batch
query.

| dbt concept | Feldera concept | Description |
| ----------------------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| `database` | _(unused)_ | Set to any string (e.g. `"default"`) |
| `schema` | Pipeline name | Each dbt schema maps to one [Feldera pipeline](https://docs.feldera.com/pipelines) (a continuously running SQL program) |
| `table` materialization | Input table | External data source (Kafka, HTTP, S3) |
| `view` materialization | View | Intermediate SQL transform inside the continuous pipeline |
Comment thread
mdrakiburrahman marked this conversation as resolved.
Outdated
| `incremental` materialization | Materialized view | IVM-backed output, also queryable via [ad-hoc queries](https://docs.feldera.com/sql/ad-hoc) for debugging |
Comment thread
mdrakiburrahman marked this conversation as resolved.
Outdated
| `seed` | Table + HTTP push | Schema registered, data pushed via HTTP ingress |

### Configuration options

Expand Down Expand Up @@ -211,6 +234,9 @@ dbt seed --full-refresh # drop and recreate, then push
## Documentation

- **[Feldera documentation](https://docs.feldera.com/)** — platform docs, SQL reference, connectors
- [Pipelines (continuous queries)](https://docs.feldera.com/pipelines) — how Feldera compiles SQL into an incremental dataflow
- [Ad-hoc queries](https://docs.feldera.com/sql/ad-hoc) — one-shot DataFusion queries for debugging materialized state
- [Materialized tables and views](https://docs.feldera.com/sql/materialized) — prerequisite for ad-hoc query access
- **[dbt documentation](https://docs.getdbt.com/)** — general dbt usage and concepts

## Contributing
Expand Down