16.20 - UTF8 Multibyte Sequences - Teradata Vantage NewSQL Engine

Teradata Vantageā„¢ NewSQL Engine International Character Set Support

prodname
Teradata Database
Teradata Vantage NewSQL Engine
vrm_release
16.20
created_date
March 2019
category
Configuration
User Guide
featnum
B035-1125-162K

To determine the length of a byte sequence in UTF8, examine the first byte.

IF the... THEN the sequence is...
high order bit is zero one byte long.

This leaves seven bits to encode information.

If a character has a Unicode value that can be represented in seven bits, it is represented as a byte containing the Unicode value. For example, Unicode value 0x0041 is transformed to UTF8 byte 0x41.

three high order bits are 110 a two-byte sequence.

The second byte has the two high order bits set to 10. There are five free bits in the first byte and six free bits in the second byte. This allows eleven bits to represent a numeric value.

If a character has a Unicode value that can be represented in eleven bits, and cannot be represented by a shorter UTF8 sequence, then it is represented as two bytes, where the free bits contain the Unicode value. For example, Unicode value 0x03F1 is transformed to UTF8 byte sequence 0xCF 0xB1.

four high order bits are 1110 a three byte sequence.

The second and third bytes have the two high order bits set to 10. There are four free bits in the first byte and six free bits in each of the second and third bytes. This allows sixteen bits to represent a numeric value.

If a character has a Unicode value that can be represented in sixteen bits, and can not be represented by a shorter UTF8 sequence, it is represented as three bytes, where the free bits contain the Unicode value (for example, Unicode value 0x3000 is transformed to UTF8 byte sequence 0xE3 0x80 0x80).