The schema of the output table depends on the value of the OutputByWord argument.
TextTokenizer Output Table Schema, OutputByWord ('true')
Column Name |
Data Type |
Description |
accumulate_column
|
Same as in input table |
Column copied from the input table. |
sn |
VARCHAR |
Serial number, in the input text, of the extracted token. |
token |
VARCHAR |
Extracted token. |
TextTokenizer Output Table Schema, OutputByWord ('false')
Column Name |
Data Type |
Description |
accumulate_column
|
Same as in input table |
Column copied from the input table. |
token |
VARCHAR |
Tokenized text, separated by the delimiter specified by the OutputDelimiter argument. |