Teradata Scoring SDK Text Tokenizer Additional Notes - Teradata Vantage

Machine Learning Engine Analytic Function Reference

Product
Teradata Vantage
Release Number
9.02
9.01
2.0
1.3
Published
February 2022
Language
English (United States)
Last Update
2022-02-10
dita:mapPath
rnn1580259159235.ditamap
dita:ditavalPath
ybt1582220416951.ditaval
dita:id
B700-4003
lifecycle
previous
Product Category
Teradata Vantageā„¢

If the function uses a dictionary table, specify it with the ModelTable and ModelTag syntax elements of AMLGenerator. If the function uses a dictionary file or CRF model file, specify it with the InstalledFile and InstalledFileTag syntax elements of AMLGenerator. If you specify no tables, the function uses the default embedded dictionaries for English or Chinese text.

Example 1: Chinese

InstalledFile ('crf_model_file')
InstalledFileTag ('CRF')

Example 2: Chinese

ModelTable ('dict_table')
ModelTableTag ('DICT')
InstalledFile ('crf_model_file')
InstalledFileTag ('CRF')
InstalledFile2 ('dict_file')
InstalledFile2Tag ('DICT')

Example 3: English or Japanese

CRF file is not supported.

ModelTable('dict_table')
ModelTableTag('DICT')
InstalledFile ('dict_file')
InstalledFileTag ('DICT')