TextTokenizer Example 2: Japanese Tokenization - Teradata Vantage

Input

User dictionary file user_dict_jp.txt:

ＳＰ女子,ＳＰ女子,エスピージョシ,カスタム名詞

SQL Call 1

SELECT * FROM TextTokenizer (
  ON jp_input AS "input" PARTITION BY any
  ON jp_dict AS dict DIMENSION
  USING
  InputLanguage ('jp')
  OutputByWord ('false')
  Accumulate ('id')
  TextColumn ('txt')
  UserDictionaryFile ('user_dict_jp.txt')
) AS dt ORDER BY id;

Output 1

SQL Call 2

SELECT * FROM TextTokenizer (
  ON jp_input AS "input" PARTITION BY any
  ON jp_dict AS dict DIMENSION
  USING
  InputLanguage ('jp')
  OutputByWord ('true')
  Accumulate ('id')
  TextColumn ('txt')
) AS dt ORDER BY id;

Output 2