Package org.apache.spark.sql.connector.metric

Interface CustomTaskMetric

@Evolving public interface CustomTaskMetric
A custom task metric. This is a logical representation of a metric reported by data sources at the executor side. During query execution, Spark will collect the task metrics per partition by PartitionReader and update internal metrics based on collected metric values. For streaming query, Spark will collect and combine metrics for a final result per micro batch.

The metrics will be gathered during query execution back to the driver and then combined. How the task metrics are combined is defined by corresponding CustomMetric with same metric name. The final result will be shown up in the data source scan operator in Spark UI.

There are a few special metric names: "bytesWritten" and "recordsWritten". If the data source defines custom metrics with the same names, the metric values will also be updated to corresponding task metrics.

Since:
3.2.0

  • Method Summary

    Modifier and Type
    Method
    Description
    default CustomTaskMetric
    mergeWith(CustomTaskMetric other)
    Merges this metric with another metric of the same name, returning a new CustomTaskMetric that represents the combined value.
    String
    name()
    Returns the name of custom task metric.
    long
    value()
    Returns the long value of custom task metric.

  • Method Details

    • name

      String name()
      Returns the name of custom task metric.

    • value

      long value()
      Returns the long value of custom task metric.

    • mergeWith

      default CustomTaskMetric mergeWith(CustomTaskMetric other)
      Merges this metric with another metric of the same name, returning a new CustomTaskMetric that represents the combined value. This is called when a task reads multiple partitions concurrently (e.g., k-way merge coalescing) to produce a single task-level value before reporting to the driver.

      The default implementation returns a new metric whose value is the sum of the two values, which is correct for count-type metrics. Data sources with non-additive metrics (e.g., max, average, last-value) must override this method to provide correct merge semantics.

      Parameters:
      other - another metric with the same name to merge with
      Returns:
      a new metric representing the merged value
      Since:
      4.2.0