16.20 - Unicode Pass Through - Teradata Vantage NewSQL Engine

Teradata Vantage™ NewSQL Engine Release Summary

prodname
Teradata Database
Teradata Vantage NewSQL Engine
vrm_release
16.20
created_date
March 2019
category
Release Notes
featnum
B035-1098-162K
Unicode Pass Through is a Unicode error handling enhancement. The feature gives users the ability to allow Pass Through Characters (PTCs) to be imported into and exported from Teradata. PTCs are the set of all possible Unicode characters minus the 6.0 BMP characters supported by Teradata and the set of noncharacters. PTCs include the following:
  • Characters not supported by Teradata:
    • Characters in BMP from Unicode versions 6.1.0 to 9.0.0
    • Characters in SMP from all Unicode versions
  • Unassigned characters
  • Private use characters

The feature is enabled or disabled at the session level and is valid only for Unicode sessions. UTF8 or UTF16 sessions with the Unicode Pass Through feature enabled will allow PTCs to be used in queries and to be stored in or retrieved from Unicode columns. Noncharacters and characters with an invalid encoding form are changed to the REPLACEMENT CHARACTER (U+FFFD) which can pass through and be stored in Teradata with this feature.

By default, this feature is not enabled. In sessions where the feature is not enabled, attempts to input any unsupported 6.0 BMP Unicode code points into Teradata results in an error and the query fails.

Benefits

  • Increases the repertoire of characters available to Unicode users from 61K to 1,112K, including Emoji and other ideographs that previously could not be stored in Teradata.
  • Reduces or eliminates cases where translation errors require users to cleanse their Unicode data before submitting it to Teradata.
  • Client load filtering is simplified, therefore improving load performance.
  • Data is not lost as would be the case when replacement characters are used.
  • Enhances Teradata Unicode support to be compatible with Unicode provided by other data sources including other databases.

Considerations

  • This feature only applies to Unicode data and sessions, meaning UTF8 and UTF16 sessions and the UNICODE server character set. For example, importing the 0x1A replacement character from an ASCII character set to a Unicode column will still be rejected even in a session enabled with the Unicode Pass Through feature.
  • Although this feature allows the storage and retrieval of PTCs, full support is not available for these characters in Teradata. For example, collation, case sensitivity, and object name support are not included with this feature.
  • PTCs should not be stored in a column that is part of an index.
  • This feature is not supported with standalone utilities such as FastLoad and MultiLoad. You should use Teradata Parallel Transporter to load or unload data containing PTCs.
  • Once PTCs are stored in Teradata, they may be difficult to remove. They would have to be deleted or replaced.
  • With this feature enabled, users who rely on Teradata to screen out unsupported characters or the REPLACEMENT CHARACTER (U+FFFD) can no longer rely on Teradata to do so.
  • This feature allows Teradata to support characters that were not previously supported. Therefore, data on a Teradata system with this feature enabled cannot be transported to a Teradata system that is running on an earlier release.

SQL Changes

SET SESSION CHARACTER SET UNICODE PASS THROUGH [ON|OFF]

Additional Information

For more information on Unicode Pass Through, see:
  • Teradata Vantage™ NewSQL Engine International Character Set Support, B035-1125
  • Teradata Vantage™ SQL Data Definition Language Syntax and Examples, B035-1144