| Argument | Category | Description |
|---|---|---|
| TextColumn | Required | Specifies the name of the input table column that contains the text to tokenize. |
| Language | Optional | Specifies the language of the text in text_column:
|
| Model | Optional | Specifies the name of model file that the function uses for tokenizing. The model must be a conditional random-fields model and model_file must already be installed on the database. If you omit this argument, or if model_file is not installed on the database, then the function uses white spaces to separate English words and an embedded dictionary to tokenize Chinese text. If you specify Language('jp'), the function ignores this argument.
|
| OutputDelimiter | Optional | Specifies the delimiter for separating tokens in the output. The default value is slash (/). |
| OutputByWord | Optional | Specifies whether to output one token in each row. The default value is 'false' (output one line of text in each row). |
| Accumulate | Optional | Specifies the names of the input table columns to copy to the output table. |
| UserDictionaryFile | Optional | Specifies the name of the user dictionary to use to correct results specified by the model. If you specify both this argument and a dictionary table (dict), then the function uses the union of user_dictionary_file and dict as its dictionary. Input describes the format of user_dictionary_file and dict. If the function finds more than one matched term, it selects the longest term for the first match.
|