Skip to content

iceberg.md

Latest commit

 

History

History
161 lines (127 loc) · 5.02 KB

iceberg.md

File metadata and controls

161 lines (127 loc) · 5.02 KB
Raw

Iceberg source (contrib)

Description

Iceberg data sources are tables managed by any supported Iceberg catalog. The IcebergSource class provides a unified interface with a configurable catalog_type parameter:

  • "rest" (default): Apache Iceberg REST Catalog specification — Unity Catalog, Apache Polaris, Nessie, Snowflake Open Catalog
  • "hive": Hive Metastore catalog
  • "glue": AWS Glue catalog
  • "sql": SQL-based (JDBC) catalog
  • "dynamodb": DynamoDB-based catalog

The data source carries catalog connection details (catalog_type, endpoint, warehouse, namespace, table, authentication). When the offline store (DuckDB, Spark) encounters this source, it resolves table metadata and credentials via the configured catalog at query time.

Examples

IcebergSource (REST catalog)

Works with any Iceberg REST Catalog:

from feast.infra.data_sources.contrib.iceberg_catalog import IcebergSource

my_source = IcebergSource(
    catalog_type="rest",  # default
    endpoint="http://localhost:8081/api/2.1/unity-catalog/iceberg",
    warehouse="unity",
    namespace="default",
    table="driver_features",
    timestamp_field="event_timestamp",
    token_env_var="UC_TOKEN",
)

IcebergSource (Hive Metastore)

from feast.infra.data_sources.contrib.iceberg_catalog import IcebergSource

my_source = IcebergSource(
    catalog_type="hive",
    catalog_properties={"uri": "thrift://metastore:9083"},
    warehouse="my_warehouse",
    namespace="default",
    table="driver_features",
    timestamp_field="event_timestamp",
)

IcebergSource (AWS Glue)

from feast.infra.data_sources.contrib.iceberg_catalog import IcebergSource

my_source = IcebergSource(
    catalog_type="glue",
    catalog_properties={"region_name": "us-east-1"},
    warehouse="my_account",
    namespace="my_database",
    table="driver_features",
    timestamp_field="event_timestamp",
)

UnityCatalogSource (with governance) {#unity-catalog-source}

Extends IcebergSource with Unity Catalog governance:

from feast.infra.data_sources.contrib.iceberg_catalog import (
    UnityCatalogSource,
)

my_uc_source = UnityCatalogSource(
    warehouse="production",
    namespace="ml_features",
    table="driver_stats",
    timestamp_field="event_timestamp",
    register_as_feature_table=True,  # Register in UC on feast apply
    sync_lineage=True,               # Record lineage in UC
)

When endpoint is omitted, it defaults to {DATABRICKS_HOST}/api/2.1/unity-catalog/iceberg. When token_env_var is omitted, it defaults to DATABRICKS_TOKEN.

Full Feature View Example

from datetime import timedelta

from feast import Entity, FeatureView, Field
from feast.types import Float64, Int64

from feast.infra.data_sources.contrib.iceberg_catalog import (
    UnityCatalogSource,
)

driver = Entity(name="driver_id", join_keys=["driver_id"])

driver_stats_source = UnityCatalogSource(
    warehouse="production",
    namespace="ml_features",
    table="driver_hourly_stats",
    timestamp_field="event_timestamp",
    created_timestamp_column="created",
)

driver_stats_fv = FeatureView(
    name="driver_hourly_stats",
    entities=[driver],
    source=driver_stats_source,
    schema=[
        Field(name="conv_rate", dtype=Float64),
        Field(name="acc_rate", dtype=Float64),
        Field(name="avg_daily_trips", dtype=Int64),
    ],
    ttl=timedelta(days=1),
    online=True,
)

Configuration Reference

IcebergSource

Parameter Type Description
catalog_type str Catalog backend: "rest" (default), "hive", "glue", "sql", "dynamodb"
endpoint str Catalog endpoint URL (required for "rest", optional for others)
warehouse str Catalog/warehouse name
namespace str Schema/namespace within the catalog
table str Table name
catalog_properties dict Additional catalog-specific properties passed to PyIceberg
timestamp_field str Event timestamp column for point-in-time joins
created_timestamp_column str Optional column indicating row creation time
token_env_var str Environment variable name holding the auth token
credential_vending bool Whether to request scoped credentials (default: True)
field_mapping dict Column name mapping from source to feature names

UnityCatalogSource (additional parameters)

Parameter Type Description
register_as_feature_table bool Register as UC feature table on feast apply (default: True)
sync_lineage bool Sync lineage metadata to Unity Catalog (default: True)

Supported Types

Iceberg Type Feast Type
boolean BOOL
int INT32
long INT64
float FLOAT
double DOUBLE
string STRING
binary BYTES
timestamp / timestamptz INT64
decimal DOUBLE
uuid STRING