TextTokenizer Arguments - Aster Analytics

Teradata AsterĀ® Analytics Foundation User GuideUpdate 2

Product
Aster Analytics
Release Number
7.00.02
Published
September 2017
Language
English (United States)
Last Update
2018-04-17
dita:mapPath
uce1497542673292.ditamap
dita:ditavalPath
AA-notempfilter_pdf_output.ditaval
dita:id
B700-1022
lifecycle
previous
Product Category
Software
TextColumn
Specifies the name of the input table column that contains the text to tokenize.
Language
[Optional] Specifies the language of the text in text_column:
  • 'en': English (Default)
  • 'zh_CN': Simplified Chinese
  • 'zh_TW': Traditional Chinese
  • 'jp': Japanese
Model
[Optional] Specifies the name of model file that the function uses for tokenizing. The model must be a conditional random-fields model and model_file must already be installed on the database. If you omit this argument, or if model_file is not installed on the database, then the function uses white spaces to separate English words and an embedded dictionary to tokenize Chinese text.
If you specify Language('jp'), the function ignores this argument.
OutputDelimiter
[Optional] Specifies the delimiter, a string, for separating tokens in the output. Default: '/' (slash).
OutputByWord
Specifies whether to output one token in each row. Default: 'false' (output one line of text in each row).
Accumulate
[Optional] Specifies the names of the input table columns to copy to the output table.
UserDictionaryFile
[Optional] Specifies the name of the user dictionary to use to correct results specified by the model. If you specify both this argument and a dictionary table (dict), then the function uses the union of user_dictionary_file and dict as its dictionary. Input describes the format of user_dictionary_file and dict.
If the function finds more than one matched term, it selects the longest term for the first match.