2.10 - Spark SQL Data Type Mapping - Teradata QueryGrid

Teradata® QueryGrid™ Installation and User Guide

prodname
Teradata QueryGrid
vrm_release
2.10
created_date
September 2019
category
Administration
Configuration
Installation
User Guide
featnum
B035-5991-099K

Global Types to Spark SQL Data Types

Global Data Types denoted with an asterisk (*) are only available with Teradata Database 16.20 Feature Update 1 and later.
Global Data Type Spark SQL Data Type
G_Array array
G_Array_VC_UTF16 / G_Array_VC_Latin * array
G_BigInt bigint
G_Blob binary
G_Boolean boolean
G_Byte binary
G_ByteInt tinyint
G_Char_Latin
Latin data type mapping is only for data types using ISO_8859_1 or US_ASCII encoding.
string
G_Char_UTF16 string
G_Clob_Latin
Latin data type mapping is only for data types using ISO_8859_1 or US_ASCII encoding.
string
G_Clob_UTF16 string
G_Date date
G_Decimal decimal
G_Double double
G_Float float
G_Integer integer
G_JSON_UTF16 / G_JSON_Latin * string
G_Map map
G_Number decimal
G_Row struct
G_SmallInt smallint
G_STGeometry * string
G_TimeStamp timestamp
G_Varbyte binary
G_Varchar_Latin
Latin data type mapping is only for data types using ISO_8859_1 or US_ASCII encoding.
string
G_Varchar_UTF16 string
G_XML * string
Others Currently not supported

Spark SQL Data Types to Global Types

Spark SQL Data Type Global Data Type
array G_Array
bigint G_BigInt
binary G_Blob
boolean G_Boolean
char G_Char_Latin
Latin data type mapping is only for data types using ISO_8859_1 or US_ASCII encoding.
char G_Char_UTF16
date G_Date
decimal G_Decimal
double G_Double
float G_Float
integer G_Integer
map G_Map
smallint G_SmallInt
string G_Clob_Latin
Latin data type mapping is only for data types using ISO_8859_1 or US_ASCII encoding.
string G_Clob_UTF16
struct G_Row
timestamp G_TimeStamp
tinyint G_ByteInt
varchar G_Varchar_Latin
Latin data type mapping is only for data types using ISO_8859_1 or US_ASCII encoding.
varchar G_Varchar_UTF16

Spark SQL String and Binary Types Considerations

Spark SQL String and Binary columns are restricted to a maximum 1GB in size. However, due to the in-memory nature of these types, a large amount of resources are required on the Spark side when their size nears the 1GB restriction. Therefore, caution is advised when inserting large Teradata CLOB or BLOB columns into the Spark SQL String or Binary columns when using QueryGrid.