Package org.apache.spark.sql.types

Class FileType

Object
org.apache.spark.sql.types.AbstractDataType
org.apache.spark.sql.types.DataType
org.apache.spark.sql.types.FileType
All Implemented Interfaces:
Serializable, scala.Equals, scala.Product
public class FileType extends DataType implements scala.Product, Serializable
The data type representing a reference to a file, with metadata describing the file.

In SQL the type is named FILE. A table column must be declared with the EXTERNAL keyword, as in CREATE TABLE t (f FILE EXTERNAL), while the signature of a function declares a bare FILE. Its JSON representation is "file external" for FileType.external and "file unknown" for FileType(), except that an external file in a struct field is the plain "file". A bare "file" is read as external inside a struct field and as unknown elsewhere.

FileType is planned to be physically stored as a struct with the following fields (see FileType.FIELDS), but it deliberately does not extend StructType: - uri (STRING): the URI of the referenced file, which may be absolute or relative - offset (BIGINT): the start of the referenced byte range - size (BIGINT): the length of the referenced byte range - content_type (STRING): the MIME/content type of the referenced bytes - checksum (STRING): an algorithm-tagged integrity token for the referenced bytes, e.g. "MD5:{digest}", or "ETAG:{etag}" for the whole file - inline (BINARY): the referenced bytes stored inline in the value

FileType may be EXTERNAL (simple references, lifecycle independent of the table) or UNKNOWN (mode not yet determined, such as the result of a cast or of a UDF). Mode is part of type identity - files with different modes are distinct under equals.

FileType is not comparable: it defines neither equality nor an ordering, as it is not clear what equality of two files should mean. These uses of a FILE fail at analysis: a join, grouping, sort or window key, DISTINCT, INTERSECT, EXCEPT and UNION without ALL, the keys of dropDuplicates, a repartition or REBALANCE key, hash functions such as hash and xxhash64, the key of a map built in a query or of a map in the return type of a function, and a partition, bucket, clustering or write-ordering column of a table. Aggregates that deduplicate or hash their input, such as count(DISTINCT f), collect_set, approx_count_distinct and mode, are not rejected yet. Accessing the fields of a FILE, such as uri, is planned together with its physical representation.

param: mode the mode for this FileType column

Since:
4.4.0
See Also:

  • Constructor Details

  • Method Details

    • external

      public static FileType external()
      Creates an external FileType.

    • URI_INDEX

      public static int URI_INDEX()
      Field index for the uri field.

    • OFFSET_INDEX

      public static int OFFSET_INDEX()
      Field index for the offset field.

    • SIZE_INDEX

      public static int SIZE_INDEX()
      Field index for the size field.

    • CONTENT_TYPE_INDEX

      public static int CONTENT_TYPE_INDEX()
      Field index for the content_type field.

    • CHECKSUM_INDEX

      public static int CHECKSUM_INDEX()
      Field index for the checksum field.

    • INLINE_INDEX

      public static int INLINE_INDEX()
      Field index for the inline field.

    • FIELDS

      public static StructField[] FIELDS()
      The struct fields that define the physical layout of FileType. Positional consumers (the *_INDEX constants) depend on this order; do not reorder without updating every layer. inline is internal: it is not exposed as a user-facing field.
      Returns:
      (undocumented)

    • PUBLIC_FIELDS

      public static StructField[] PUBLIC_FIELDS()
      The user-facing subset of `FIELDS`: everything except `inline`.

    • asStruct

      public static StructType asStruct()
      Returns the FileType fields as a StructType, for use in expressions.

    • mode

      public FileTypeMode mode()

    • defaultSize

      public int defaultSize()
      Description copied from class: DataType
      The default size of a value of this data type, used internally for size estimation.
      Specified by:
      defaultSize in class DataType
      Returns:
      (undocumented)

    • typeName

      public String typeName()
      Description copied from class: DataType
      Name of the type used in JSON serialization.
      Overrides:
      typeName in class DataType