Feldera and Databricks Enzyme Demo by anandbraman · Pull Request #6612 · feldera/feldera

anandbraman · 2026-07-09T22:22:10Z

Describe Manual Test Plan

Checklist

Unit tests added/updated
Integration tests added/updated
Documentation updated
Changelog updated

Breaking Changes?

Mark if you think the answer is yes for any of these components:

OpenAPI / REST HTTP API / feldera-types / manager (What is a breaking change?)
Feldera SQL (Syntax, Semantics)
feldera-sqllib (incl. dependencies fxp, etc.) (What is a breaking change?)
Python SDK (What is a breaking change?)
fda (CLI arguments)
Adapters (including configuration)
Storage Format / Checkpoints
Others (specify)

Describe Incompatible Changes

mihaibudiu · 2026-07-09T22:36:54Z


 With Feldera, you use the same SQL your batch engine runs now, get the same results, and pay a cost that tracks change size instead of data size. In addition, you get millisecond update latency across your full medallion architecture.
+
+Continue on to Part 4 for a comparison between Databricks' IVM offering (Lakeflow Declarative Pipelines) and Feldera.


should this be a hyperlink?

mihaibudiu · 2026-07-09T22:39:34Z

+    return datetime.fromisoformat(value.replace("Z", "+00:00"))
+
+
+def delta_connector(inputs: list, table: str) -> dict | None:


this seems to be list[dict], but maybe you don't know the dict type?

added annotation

mihaibudiu · 2026-07-09T22:44:13Z

@@ -0,0 +1,497 @@
+# Databricks notebook source
+# MAGIC %md


I don't know what these MAGICs are

It's just the syntax that defines a cell in a notebook

mihaibudiu · 2026-07-09T22:44:48Z

+# MAGIC - **Silver / Gold** are `@dp.materialized_view`s, persisted in the pipeline's Unity
+# MAGIC   Catalog target schema.
+# MAGIC
+# MAGIC The business logic (SQL) is identical to the batch script and to the Feldera views


Is it identical or equivalent?

the business logic is identical. The sql is equivalent

mihaibudiu · 2026-07-09T22:48:07Z

@@ -0,0 +1,305 @@
+# Part 4: Compare Feldera and Databricks Lakeflow Declarative Pipelines (Enzyme)
+
+Parts [1](./part1.md)–[3](./part3.md) read the Bronze snapshot and CDC from Feldera's


I would actually list all 3 parts with a short summary for each of them and a hyperlink to make it easy for people to go there.

mihaibudiu · 2026-07-09T22:49:42Z

+| `job.yaml` | Settings for the `compute_silver_gold` job (Step 2). |
+| `measure_latency.py` | Reports Feldera's per-connector processing and end-to-end latency for changes (Step 4). |
+
+## Prerequisites


if you want to run the demo by yourself

mihaibudiu · 2026-07-09T22:52:06Z

+
+## Step 3 — `ingest_bronze` as a data-generating process
+
+Each run of the `ingest_bronze` task writes one hour of change into the Bronze Delta tables


one hour between updates is a lot... unless you are replaying an existing log very fast

yeah the user just triggers the job that points to a file i.e., "2025-11-30T00". There's no schedule for the job, it's all interactive.

mihaibudiu · 2026-07-09T22:54:36Z

+
+Feldera pays neither cost the same way. It is always on, so there is no per-run
+provisioning, and it maintains every view incrementally, so the update is proportional to
+the size of the change. Updates are reflected in less than a second for one CDC hour, not 58s.


how about the machines resources used for these two? are they the same?

Databricks is serverless, so I'm not sure exactly what the resource usage is. Feldera doesn't use more than a couple GiBs of RAM

That would be useful to find spell out. Can you guess the DBX usage based on cost?

At this scale both cost very little. I'll be doing a cost analysis for the blog post where we test multiple times at 5 or so different scale factors. Serverless is called out as a dep in the DBX set up explicitly

mihaibudiu · 2026-07-09T22:56:11Z

+| `gold_product_demand_surge` | Full recompute every refresh | Incremental |
+| Orchestration | Job schedule + pipeline refresh | None — change-driven |
+| Per-run startup | ~28s serverless provisioning | None — always on |
+| Freshness for the first CDC hour | ~86s end-to-end | 0.37s at most |


This should be a graph with variance bars on measurements: run the measurements 10 times and plot some error bars for each measurement, with us and Databricks on the same graph. The graph should be the TLDR.

I think this would be a good idea for a blog post. @ryzhyk and I are going to publish something that tests with larger scale data. I'll run multiple experiements per scale factor

mythical-fred

Two commit messages need cleanup before merge — see inline comments for details.

mythical-fred · 2026-07-10T01:00:46Z

+public demo bucket. This part deploys the Databricks assets into **your own** workspace
+and **your own** S3 bucket, then repoints the Feldera pipeline at that bucket. Two things
+run against the same Bronze Delta tables:
+


Nit: "Lakeflow Declarative Pipelies" → "Pipelines" (typo).

mythical-fred · 2026-07-10T01:01:09Z

Commits 2 and 3 (sidebar and add latency measure for feldera. provide comparison metrics for each platform) need rewriting before merge. These are lazy one-liners for non-trivial changes — the first adds a sidebar entry, the second adds an entire latency-measurement script and comparison results.

Please git rebase -i to either squash into a single well-structured commit or give each commit a proper subject + body explaining why. The first commit's message is good — use that as a model.

Git history is a first-class artifact. Resources:

https://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html

Added configuration for a Lakeflow declarative pipeline. Users can copy the bronze data to their own bucket, see how Feldera works with snapshot_and_follow mode, and contrast with Databricks IVM solution (lakeflow declarative pipelines) Signed-off-by: Anand Raman <anand.raman@feldera.com>

mythical-fred

Both blockers from my prior review are addressed — commits squashed into a single clean message, and the "Pipelies" typo is fixed. Content still reads well. Leaving Mihai's open discussion points (resource/cost breakdown for the Databricks side, variance-bar graph as a follow-up) to you two.

anandbraman requested review from lalithsuresh, ryzhyk and tristenfeldera July 9, 2026 22:22

anandbraman changed the title ~~Feldera databricks ivm demo~~ Feldera and Databricks Enzyme comparison Jul 9, 2026

anandbraman changed the title ~~Feldera and Databricks Enzyme comparison~~ Feldera and Databricks Enzyme Demo Jul 9, 2026

mihaibudiu approved these changes Jul 9, 2026

View reviewed changes

mythical-fred suggested changes Jul 10, 2026

View reviewed changes

anandbraman force-pushed the feldera-databricks-ivm-demo branch from e4e6ce2 to 5e74d7d Compare July 10, 2026 17:33

mythical-fred approved these changes Jul 10, 2026

View reviewed changes

anandbraman added this pull request to the merge queue Jul 10, 2026

Merged via the queue into main with commit b203ac3 Jul 10, 2026
1 check passed

anandbraman deleted the feldera-databricks-ivm-demo branch July 10, 2026 22:09


		With Feldera, you use the same SQL your batch engine runs now, get the same results, and pay a cost that tracks change size instead of data size. In addition, you get millisecond update latency across your full medallion architecture.

		Continue on to Part 4 for a comparison between Databricks' IVM offering (Lakeflow Declarative Pipelines) and Feldera. No newline at end of file

		return datetime.fromisoformat(value.replace("Z", "+00:00"))


		def delta_connector(inputs: list, table: str) -> dict \| None:

		@@ -0,0 +1,305 @@
		# Part 4: Compare Feldera and Databricks Lakeflow Declarative Pipelines (Enzyme)

		Parts [1](./part1.md)–[3](./part3.md) read the Bronze snapshot and CDC from Feldera's


		## Step 3 — `ingest_bronze` as a data-generating process

		Each run of the `ingest_bronze` task writes one hour of change into the Bronze Delta tables

Uh oh!

Conversation

anandbraman commented Jul 9, 2026

Describe Manual Test Plan

Checklist

Breaking Changes?

Describe Incompatible Changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mythical-fred left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mythical-fred commented Jul 10, 2026

Uh oh!

mythical-fred left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants