Commit a6d3929
[avro] Confluent JDBC sink connector Avro format.
Add support for the Avro output format compatible with Confluent
JDBC sink connector.
This format encodes primary key columns as Kafka message keys.
Inserts and updates carry the complete value of the new record
in the message payload. Deletes are represented as tombstone
messages with null values.
The connector supports all three standard subject name strategies:
topic name, record name, and topic + record name.
The connector automatically generates value schema where all non-key
columns are nullable. This helps to make generated schemas predictable,
as otherwise the user has little control over the nullability of
SQL view columns.
Here is a minimal connector config example, which should to the right
thing in most cases:
```
create view test_view
WITH (
'connectors' = '[{
"format": {
"name": "avro",
"config": {
"update_format": "confluent_jdbc",
"registry_urls": ["http://localhost:18081"],
"key_fields": ["id"]
}
},
"transport": {
"name": "kafka_output",
"config": {
"bootstrap.servers": "localhost:19092",
"topic": "avro_jdbc_test"
}
}
}
]'
)
as select * from test_table;
```
Docs and more tests are coming in a separate PR.
Signed-off-by: Leonid Ryzhyk <leonid@feldera.com>1 parent f3327e2 commit a6d3929
File tree
22 files changed
+990
-230
lines changed- crates
- adapters/src
- controller
- format
- avro
- json
- parquet
- integrated/delta_table
- static_compile
- test
- transport
- http
- kafka
- ft
- nonft
- feldera-types/src
- format
- demo/project_demo08-DebeziumJDBC
- deploy
22 files changed
+990
-230
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1459 | 1459 | | |
1460 | 1460 | | |
1461 | 1461 | | |
1462 | | - | |
| 1462 | + | |
| 1463 | + | |
| 1464 | + | |
| 1465 | + | |
| 1466 | + | |
| 1467 | + | |
1463 | 1468 | | |
1464 | 1469 | | |
1465 | 1470 | | |
| |||
1940 | 1945 | | |
1941 | 1946 | | |
1942 | 1947 | | |
1943 | | - | |
1944 | | - | |
| 1948 | + | |
| 1949 | + | |
1945 | 1950 | | |
1946 | 1951 | | |
1947 | 1952 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
140 | 140 | | |
141 | 141 | | |
142 | 142 | | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
143 | 153 | | |
144 | 154 | | |
145 | 155 | | |
| |||
230 | 240 | | |
231 | 241 | | |
232 | 242 | | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
233 | 246 | | |
234 | 247 | | |
235 | 248 | | |
| |||
357 | 370 | | |
358 | 371 | | |
359 | 372 | | |
| 373 | + | |
| 374 | + | |
| 375 | + | |
360 | 376 | | |
361 | 377 | | |
362 | 378 | | |
| |||
0 commit comments