TD_TextParser Usage Notes - Analytics Database

Database Analytic Functions

Deployment
VantageCloud
VantageCore
Edition
Enterprise
IntelliFlex
VMware
Product
Analytics Database
Release Number
17.20
Published
June 2022
ft:locale
en-US
ft:lastEdition
2025-04-01
dita:mapPath
gjn1627595495337.ditamap
dita:ditavalPath
qkf1628213546010.ditaval
dita:id
jmh1512506877710
Product Category
Teradata Vantage™
  • The 'locations' column will accommodate a list of comma separated values up to the maximum capacity of a VARCHAR 64000 column. Once this limit is reached or if the number of digits to be written exceeds it, any additional values will be ignored and not included in the output.
  • TokenColName has the following limitations when using ListPositions and TokenFrequency:
    • Cannot have spaces.
    • Do not use any reserved SQL word
  • In TokenColName, do not use column names which require the use of double quotations, e.g., containing special characters, reserved keywords, numerics only are not allowed for TokenColName. Some examples of names not allowed: '1,2,3' has numbers and special characters, '123' has only numbers, 'Order' is a reserved keyword.
  • When StemTokens(‘true’) now the function ignores ConvertToLowerCase and behaves as if it had the default value 'true'. Instead of previous behavior: Error in function TD_TextParser: ConvertToLowerCase needs to be true for stemming.
  • You cannot use Delimiter and DelimiterRegex at the same time.
  • If neither Delimiter nor DelimiterRegex are provided, default Delimiter value is used for tokenization.
  • When using DelimiterRegex, empty tokens are not part of the output and are silently discarded.