Package org.apache.spark.io

Class HadoopCodecStreams

Object

org.apache.spark.io.HadoopCodecStreams

public class HadoopCodecStreams
extends Object

An utility object to look up Hadoop compression codecs and create input streams. In addition to standard Hadoop codecs, it also supports Spark's Zstandard codec if Hadopp is not compiled with Zstandard support. Additionally, it supports non-standard file extensions like .zstd and .gzip for Zstandard and Gzip codecs.

Constructor Summary

Constructors

Constructor	Description
HadoopCodecStreams()

Method Summary

Modifier and Type	Method	Description
static InputStream	createInputStream(org.apache.hadoop.conf.Configuration config, org.apache.hadoop.fs.Path file)
static scala.Option<InputStream>	createZstdInputStream(org.apache.hadoop.conf.Configuration config, org.apache.hadoop.fs.Path file, InputStream inputStream)
static String	DECOMPRESSION_CODEC_CONF()	Hadoop Configuration key carrying the read-side compression codec explicitly requested via the decompression data source option.
static scala.Option<org.apache.hadoop.io.compress.CompressionCodec>	getDecompressionCodec(org.apache.hadoop.conf.Configuration config, org.apache.hadoop.fs.Path file)	Returns the codec to decompress file with.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

HadoopCodecStreams

public HadoopCodecStreams()

Method Details

DECOMPRESSION_CODEC_CONF

public static String DECOMPRESSION_CODEC_CONF()

Hadoop Configuration key carrying the read-side compression codec explicitly requested via the decompression data source option. The value is a full codec class name, an empty string to force "no compression" (read the file as-is), or absent to infer the codec from the file name extension. Set by the file format / scan read paths and consumed by getDecompressionCodec(org.apache.hadoop.conf.Configuration,org.apache.hadoop.fs.Path) so it uniformly overrides extension-based detection for line-based, multi-line, whole-text, and schema-inference reads.

This is an internal key, deliberately outside the spark.* namespace so it is not mistaken for a user-facing configuration. The read paths always overwrite or unset it, so a value inherited from the session or Hadoop configuration is never honored.

Returns:

(undocumented)

getDecompressionCodec

public static scala.Option<org.apache.hadoop.io.compress.CompressionCodec> getDecompressionCodec(org.apache.hadoop.conf.Configuration config,
 org.apache.hadoop.fs.Path file)

Returns the codec to decompress file with. When a read compression option has been set (see DECOMPRESSION_CODEC_CONF()) it takes precedence over the file name extension: a codec class name selects that codec, and an empty value means the file is read as-is. Otherwise the codec is inferred from the file name extension.

Parameters:

config - (undocumented)

file - (undocumented)

Returns:

(undocumented)

createZstdInputStream

public static scala.Option<InputStream> createZstdInputStream(org.apache.hadoop.conf.Configuration config,
 org.apache.hadoop.fs.Path file,
 InputStream inputStream)

createInputStream

public static InputStream createInputStream(org.apache.hadoop.conf.Configuration config,
 org.apache.hadoop.fs.Path file)