External data may arrive from an object in a compressed format. The following are compression types supported for the supported file formats:
Iceberg
Parquet | ORC | AVRO | |
---|---|---|---|
SNAPPY | Yes | Yes | Yes |
ZSTD (zstandard) | Yes | Yes | Yes |
ZLIB | No | Yes | No |
LZ4 | No | Yes | No |
GZIP (deflate) | Yes | No | Yes |
LZO | No | No | No |
ZSTD compression is the default for Iceberg, as it provide a good compression ratio without compromising the read/write performance. Snappy provides a good read/write performance; however, its compressions ratio is lower than ZSTD or GZIP. If the storage costs permits, Snappy is a better compression choice.
Delta Lake
Parquet | |
---|---|
SNAPPY | Yes |
ZSTD (zstandard) | No |
ZLIB | No |
LZ4 | No |
GZIP (deflate) | Yes |
LZO | Yes |