Documentation
¶
Overview ¶
Package debeziumio contains cross-language functionality for using Debezium (http://kafka.apache.org/). These transforms only work on runners that support cross-language transforms.
Setup ¶
Transforms specified here are cross-language transforms implemented in a different SDK (listed below). During pipeline construction, the Go SDK will need to connect to an expansion service containing information on these transforms in their native SDK.
To use an expansion service, it must be run as a separate process accessible during pipeline construction. The address of that process must be passed to the transforms in this package.
The version of the expansion service should match the version of the Beam SDK being used. For numbered releases of Beam, these expansions services are released to the Maven repository as modules. For development versions of Beam, it is recommended to build and run it from source using Gradle.
Current supported SDKs including expansion service modules ¶
Java:
- Vendored Module: beam-sdks-java-io-debezium-expansion-service
- Run via Gradle: ./gradlew :sdks:java:io:debezium:expansion-service:shadowJar java -jar <path-to-debezium-jar> <port>
- Reference Class: org.apache.beam.io.debezium.DebeziumIO
Index ¶
- func ConnectionProperties(cp []string) readOption
- func ExpansionAddr(expansionAddr string) readOption
- func MaxRecord(r int64) readOption
- func MaxTimeToRun(r int64) readOption
- func OffsetStoragePath(path string) readOption
- func Read(s beam.Scope, username, password, host, port string, ...) beam.PCollection
- func StartOffset(offset []string) readOption
- type DriverClassName
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func ConnectionProperties ¶
func ConnectionProperties(cp []string) readOption
ConnectionProperties specifies properties of the debezium connection passed as a string with format [propertyName=property;]*
func ExpansionAddr ¶
func ExpansionAddr(expansionAddr string) readOption
ExpansionAddr sets the expansion service address to use for DebeziumIO cross-langauage transform.
func MaxRecord ¶
func MaxRecord(r int64) readOption
MaxRecord specifies maximum number of records to be fetched before stop.
func MaxTimeToRun ¶ added in v2.73.0
func MaxTimeToRun(r int64) readOption
MaxTimeToRun specifies maximum number of milliseconds to run before stop.
func OffsetStoragePath ¶ added in v2.73.0
func OffsetStoragePath(path string) readOption
OffsetStoragePath sets a path where the connector offset is automatically saved after each checkpoint and loaded on pipeline startup, allowing the pipeline to resume from where it left off without any manual offset management.
The path can be on any filesystem supported by the active Beam runner (local disk, GCS, S3, etc.).
Example:
debeziumio.OffsetStoragePath("gs://my-bucket/debezium/orders-offset.json")
When set, takes precedence over StartOffset.
func Read ¶
func Read(s beam.Scope, username, password, host, port string, connectorClass DriverClassName, t reflect.Type, opts ...readOption) beam.PCollection
Read is an external PTransform which reads from Debezium and returns a JSON string. It requires the address of an expansion service for Debezium IO. If both the host and port address are provided as "", an appropriate expansion service will be automatically started; however this is slower than having a persistent expansion service running.
Example:
username := "debezium"
password := "dbz"
host := "localhost"
port := "5432"
connectorClass := debeziumIO.POSTGRESQL
maxrecords := 1
debeziumio.Read(s.Scope("Read from debezium"), expansionAddr, username, password, host, port, connectorClass,
reflectx.String, debeziumio.MaxRecord(maxrecords), debeziumio.ExpansionAddr("localhost:9000"))
func StartOffset ¶ added in v2.73.0
func StartOffset(offset []string) readOption
StartOffset specifies the offset from which the connector should resume consuming changes. Each entry must be a "key=value" string, where numeric values are encoded as their decimal string representation.
Example for PostgreSQL:
debeziumio.StartOffset([]string{"lsn=28160840"})
Example for MySQL:
debeziumio.StartOffset([]string{"file=binlog.000001", "pos=156"})
Obtain the offset from the output of a previous pipeline run. Numeric values such as LSN or binlog position are automatically parsed to Long on the Java side.
Types ¶
type DriverClassName ¶
type DriverClassName string
DriverClassName is the type for valid and supported Database connectors for Debezium IO.
const ( // MySQL connector for Debezium MySQL DriverClassName = "MySQL" // PostgreSQL connector for Debezium PostgreSQL = "PostgreSQL" )
Source Files