About Parquet Format with Native Object Store | NOS | Teradata Vantage - 17.05 - About Parquet Format - Advanced SQL Engine - Teradata Database

Teradata Vantageā„¢ - Native Object Store Getting Started Guide

Product
Advanced SQL Engine
Teradata Database
Release Number
17.05
Release Date
January 2021
Content Type
Programming Reference
Publication ID
B035-1214-175K
Language
English (United States)

When reading external Parquet data, the maximum Parquet page size supported is 16 MB.

Using NOS, the maximum record size is 16,776,192 bytes.

  • If your record consists of all character data, these are the limitations for each character set:
    • For UNICODE, 16,776,192 bytes is equivalent to 8,388,096 characters.
    • For LATIN, 16,776,192 bytes is equivalent to 16,776,192 characters.

If some of data in the record is binary, the maximum number of characters is proportionately reduced.

Unlike CSV and JSON, Parquet tables don't have a payload column. The user creates a foreign table and maps the Parquet logical data type to the corresponding Teradata data type.

Redshift is the supported format for the manifest files.

Parquet format limitations:

  • The READ_NOS table operator does not support Parquet. However, READ_NOS can be used to view the Parquet schema, using RETURNTYPE('NOSREAD_PARQUET_SCHEMA'). This is helpful in creating the foreign table when you do not know the schema of your Parquet data beforehand.
  • Certain complex data types are not supported, including STRUCT, MAP, LIST, and ENUM.
  • Because support for the STRUCT data type is not available, nested Parquet object stores cannot be processed by NOS.

The following examples use external Parquet data in this format:

message schema {
  optional double GageHeight2;
  optional double Flow;
  optional int64 site_no;
  optional binary datetime (UTF8);
  optional double Precipitation;
  optional double GageHeight;
}

For supported Parquet formats, see Parquet External Files.