Co-authored-by: Cursor <cursoragent@cursor.com>

Exclude _timestamp/_updating_meta from Delta catalog and Parquet schema,
cast user columns to Delta-compatible types, and support multiple
timestamp units in watermark max extraction.

Co-authored-by: Cursor <cursoragent@cursor.com>

Planner may evaluate watermark expressions to millisecond (or other)
timestamp arrays after projection; unify max extraction across units
instead of requiring TimestampNanosecondArray only.

Co-authored-by: Cursor <cursoragent@cursor.com>

deltalake-aws 0.15+ requires either a DynamoDB lock client or
AWS_S3_ALLOW_UNSAFE_RENAME=true. Function-Stream sinks are single-writer,
so default the flag on; users can opt into ETag conditional puts via the
s3.conditional.put option (e.g. MinIO).

Co-authored-by: Cursor <cursoragent@cursor.com>

Replace the bare AWS_S3_ALLOW_UNSAFE_RENAME default with an explicit
DeltaCommitStrategy abstraction: single-writer (default, leveraging the
pipeline scheduler's per-table writer guarantee) or DynamoDB (for
multi-process writers). Adds delta.commit.strategy and
delta.dynamodb.* sink options for clear cloud/on-prem deployments.

Co-authored-by: Cursor <cursoragent@cursor.com>

- AmazonS3Builder now uses an exponential-backoff RetryConfig (100ms..5s,
  5 retries, 30s overall) so transient network blips don't fail commits.
- Custom S3 endpoints force path-style addressing on both the object_store
  client and the deltalake storage options (AWS_VIRTUAL_HOSTED_STYLE_REQUEST
  =false), matching MinIO/Ceph/R2/private gateway deployments.
- Preflight bucket check now surfaces InvalidAccessKeyId, SignatureDoes
  NotMatch and AccessDenied as clear configuration errors.
- on_close logs commit failures with error! before bubbling up to make
  data-loss risk visible in operator dashboards.

Co-authored-by: Cursor <cursoragent@cursor.com>

Introduce physical_sink module with FinishedFile contract,
SingleThreadPhysicalSink (S3/MinIO encode+PUT without _delta_log),
and LocalPhysicalSink. DeltaSinkOperator buffers batches, delegates
early flush to the physical layer, and commits via DeltaTableCommitter
only at checkpoint boundaries.

Co-authored-by: Cursor <cursoragent@cursor.com>

…oughput fixes

Add MQTT, HTTP, ROS/ROS2, and robot-bag connectors with ROS1 bag and PCL PCD
parsing. Persist auto-generated Kafka consumer group IDs at CREATE TABLE.
Remove hot-path debug logging from Kafka source and Delta sink, and tune
Delta/Kafka batching for higher throughput. Add bilingual source docs.

Co-authored-by: Cursor <cursoragent@cursor.com>

Loop Arrow JSON decode/flush and concat batches so source batches up to
4096 rows align with valid_indices, fixing SO101 collector DEGRADED state.

Co-authored-by: Cursor <cursoragent@cursor.com>