Skip to content

README.md

Latest commit

 

History

History
35 lines (25 loc) · 1.52 KB

README.md

File metadata and controls

35 lines (25 loc) · 1.52 KB

Dataflow flex templates

Open in Cloud Shell

Samples showing how to create and run an Apache Beam template with a custom Docker image on Google Cloud Dataflow.

Streaming Beam SQL sample (basic)

streaming_beam_sql

This sample shows how to deploy an Apache Beam streaming pipeline that reads JSON encoded messages from Pub/Sub, uses Beam SQL to transform the message data, and writes the results to a BigQuery table.

Kafka to BigQuery sample (advanced)

kafka_to_bigquery

This sample shows how to deploy an Apache Beam streaming pipeline that reads JSON encoded messages from Apache Kafka, decodes them, and writes them into a BigQuery table.

For this, we need two parts running:

  1. A Kafka service container accessible through an external IP address. This services publishes messages to a topic.
  2. An Apache Beam streaming pipeline running in Dataflow Flex Templates. This subscribes to a Kafka topic, consumes the messages that are published to that topic, processes them, and writes them into a BigQuery table.