Parquet Format with Native Object Store | NOS | Teradata Vantage - 17.10 - Parquet Format - Advanced SQL Engine - Teradata Database

Teradata Vantage™ - Native Object Store Getting Started Guide

Product
Advanced SQL Engine
Teradata Database
Release Number
17.10
Release Date
July 2021
Content Type
Programming Reference
Publication ID
B035-1214-171K
Language
English (United States)

When reading external Parquet data, the maximum Parquet page size supported is 16 MB.

Using NOS, the maximum record size is 16,776,192 bytes.
  • If your record consists of all character data, these are the limitations for each character set:
    • For UNICODE, 16,776,192 bytes is equivalent to 8,388,096 characters.
    • For LATIN, 16,776,192 bytes is equivalent to 16,776,192 characters.

If some of data in the record is binary, the maximum number of characters is proportionately reduced.

Parquet tables don't have a payload column. The user creates a foreign table and maps the Parquet logical data type to the corresponding Teradata data type.

Redshift is the supported format for the manifest files.

Parquet format limitations:
  • READ_NOS can be used to view the Parquet schema, using RETURNTYPE('NOSREAD_SCHEMA'). This is helpful in creating the foreign table when you do not know the schema of your Parquet data beforehand.
    If you are using the FULLSCAN option, continue to use NOSREAD_PARQUET_SCHEMA because FULLSCAN is not available in NOSREAD_SCHEMA.
  • Certain complex data types are not supported, including STRUCT, MAP, LIST, and ENUM.
  • Because support for the STRUCT data type is not available, nested Parquet object stores cannot be processed by NOS.

The following examples use external Parquet data in this format:

message schema {
  optional double GageHeight2;
  optional double Flow;
  optional int64 site_no;
  optional binary datetime (UTF8);
  optional double Precipitation;
  optional double GageHeight;
}

For supported Parquet formats, see Parquet External Files.