VantageCloud Lake Datalake support includes support for the following.
Supported Open Table Format
- Apache Iceberg version 2
- Delta Lake version 3
Supported External Data File Formats
For Apache Iceberg version 2:
- Parquet
- Optimized Row Columnar (ORC)
- Avro
For Delta Lake version 3:
- Parquet
Supported Catalogs
For Apache Iceberg version 2:
- Hive Metastore (HMS)
- Amazon Glue
- Unity
For Delta Lake version 3:
- Amazon Glue
- Unity
Supported Operations per Catalog
The following operations apply to both AWS and Azure.
For Apache Iceberg version 2:
- AWS Glue/Hive/Unity: READ
- AWS Glue/Hive: WRITE
For Delta Lake version 3:
- AWS Glue/Unity: READ
- Unity: WRITE
Supported Catalog and Storage Credentials
- AWS: IAM roles and assume role policies (Teradata recommends use the latter with external users on AWS).
- Azure: Service Principle-based authentication
- Unity and Databricks: Azure/Entra/AD unmanaged authorization (tenantId, client key, and client secret)
Supported Object Stores
For Apache Iceberg version 2:
- Amazon Simple Storage Service (S3)
- Azure Data Lake Storage Gen2
- Microsoft Azure BLOB Storage
For Delta Lake version 3:
- Amazon Simple Storage Service (S3)
- Azure Data Lake Storage Gen2
Supported Compression Formats
External data may be in a compressed format.
For Apache Iceberg version 2:
| Compression Format | Parquet | ORC | Avro |
|---|---|---|---|
| Snappy | Supported | Supported | Supported |
| zstd (Zstandard) | Supported | Supported | Supported |
| zlib | Unsupported | Supported | Unsupported |
| LZ4 | Unsupported | Supported | Unsupported |
| gzip (DEFLATE) | Supported | Unsupported | Supported |
For Delta Lake version 3, the following are supported for Parquet:
- Snappy
- LZO
- gzip (DEFLATE)
Supported Encryption
- Transport Layer Security (TLS) 1.2
Data is transmitted between VantageCloud Lake and the external object storage using TLS 1.2 whether or not the data is encrypted in the external object storage.
Supported Iceberg Data Types
| Iceberg Data Type | Corresponding VantageCloud Lake Data Type |
|---|---|
| boolean | BYTEINT |
| integer | INTEGER |
| long | BIGINT |
| float | DOUBLE PRECISION/FLOAT/REAL |
| double | DOUBLE PRECISION/FLOAT/REAL |
| date | DATE |
| time | TIME(6) |
| timestamp | TIMESTAMP(6) |
| timestamptz | TIMESTAMP(6) WITH TIME ZONE |
| string | VARCHAR(32000) |
| uuid | VARBYTE(32) |
| fixed(L) | L <= 64000: BYTE(L) L > 64000: BLOB |
| binary | BLOB Default maximum size: 2 GB |
| decimal | DECIMAL(precision, scale) |
| list | VARCHAR(32000) |
| map | VARCHAR(32000) |
| struct | VARCHAR(32000) |
Supported Delta Lake Data Types
| Delta Lake Data Type | Corresponding VantageCloud Lake Data Type |
|---|---|
| BIGINT or LONG | BIGINT |
| BINARY | BLOB Default maximum size: 2 GB |
| BOOLEAN | BYTEINT |
| DATE | DATE |
| DECIMAL(p,s) | DECIMAL(precision,scale) |
| DOUBLE | DOUBLE PRECISION/FLOAT/REAL |
| FLOAT | DOUBLE PRECISION/FLOAT/REAL |
| INT | INTEGER |
| SMALLINT | SMALLINT |
| STRING | VARCHAR(32000) UNICODE |
| TIMESTAMP | TIMESTAMP(6) |
| TIMESTAMP_NTZ | TIMESTAMP(6) |
| TINYINT or BYTE | BYTEINT |
| ARRAY | VARCHAR(32000) UNICODE |
| MAP | VARCHAR(32000) UNICODE |
| STRUCT | VARCHAR(32000) UNICODE |