The following limitations affect the use of Spark-to-Teradata links:
- Transaction semantics between systems is not supported.
- Import of temporal tables is not supported.
- QueryGrid does not collect query metrics (such as CPU usage) for remote Teradata queries.
- The maximum size supported for BLOB and CLOB is less than 2GB (2,097,088,000).
- The maximum size of VARCHAR is 64k.
- The temporary database name, NVP, is not supported on Teradata Database version 15.10.
- The Foreign Function Execution (FFE) feature is not supported for the Spark SQL target connector.
- The Spark connector does not support ACID tables or transactional tables.
After data has been exported and committed to a remote system, any subsequent errors or aborts on the local system do not roll back the remote request.
- When using the EXPLAIN command with a Spark initiator connector, the remote server does not return query and execution plan data.
- The default for Timestamp precision is nine (9); QueryGrid truncates data with more than six decimal places when using Spark-to-Teradata links.
- Only limited predicate pushdown is available.
- When starting either the Spark Thrift Server or Spark Shell to use with the Spark Connector, Teradata recommends setting the spark.task.maxFailures property to 1 and the spark.blacklist.enabled property to false.
- The following are a result of possible Apache Spark limitations:
- Spark 2.1 and later: When using the Spark initiator, if the schema of a target table changes after a non-native table representing that target table has been created, that non-native table must be recreated in order to reflect the schema change.
- Spark 2.2 and later: When importing data for the DATE type using the Spark target connector or exporting data of the DATE type using the Spark initiator, the data value from Spark can be incorrect.
- Spark 2.2 and later: Spark does not support Char/Varchar; therefore, when using the Spark target connector to insert data from QueryGrid and the target table contains char/varchar columns, the data from QueryGrid may be incorrect. To avoid possible incorrect data, use String instead of Char/Varchar.
- If Spark is upgraded or the location of standard Spark JARs are changed, a tdqg-node restart is required.
- UTF-16 supplementary characters longer than 2 bytes in a table cause data truncation.
- IMPORT is not supported on the VARCHAR, STRING, and CHAR columns of the Spark table if the table character set is something other than Latin or UTF-16.
- Condition pushdown of the LIMIT clause is not supported. The Spark SQL initiator SHOW command may display a limited number of rows, but all rows are imported.
- Date literals used in WHERE clauses are not converted to the time zone of the remote system if the remote system time zone is different from the initiator system time zone.
- Comparisons with DATE in the WHERE clause may yield incorrect results.
- When running a SELECT query, such as displaying the results on the Spark Shell, that is expected to transmit a large amount of rows from Teradata,Teradata recommends adding a LIMIT clause to the query if possible.