This epic is the umbrella epic where I try and capture all planned/on-going and tracked work related to the datafusion integration.

Vortex Issues

• DataFusion's casting semantics incompatibilities are causing runtime errors #6269
• Report constant columns back to DataFusion as EquivalenceProperties constant_exprs #1507
• Output ordering support (reverting Don't allow scanning tables with defined output ordering #2429)
• Respecting more global configurations (or explicitly error if they are used), and making more vortex-specific things configurable.
  • FileSinkConfig::table_partition_cols and FileSinkConfig::keep_partition_by_columns (Not sure if that's still relevant)
• Once released in DataFusion, support decimal32/64 conversion to Arrow. Convert decimal dtype to arrrow's decimal32/64 #4668
• Support Dynamic filter expressions Support datafusion dynamic filter expressions #4034
• Metadata table function - Add a vortex_metadata table UDF to display metadata in Datafusion #5144

DataFusion issues we filed

• Merging Statistics is slow when sum statistic is present apache/datafusion#15809
• FileFormat API makes it hard to avoid round trips apache/datafusion#13720 (I think FileMetadataCache might've solved this issue, need to make sure)
• Should can_coerce_from promote Utf8 to LargeUtf8? apache/datafusion#13325 (Potentially solved)
• TPC-DS query 76 becomes much slower as target_partitions goes up apache/datafusion#20078
• Optimize PhysicalExprSimplifier apache/datafusion#20111
• Make session setup an extension point for the CLI apache/datafusion#21972

DataFusion issues we're interested in

• Support Push down expression evaluation in TableProviders apache/datafusion#14993
• [EPIC] A collection of support for metadata columns in ListingTable apache/datafusion#20135

Related

• If there's willingness, support Vortex in datafusion-dft to have an easy demo Support Vortex as a file format datafusion-contrib/datafusion-dft#336

Older solved issues

• Migrating from DataSink to FileSink once the next major release (45) goes out.
• FileScanConfig::limit
• FileScanConfig::table_partition_cols