1.0 - 8.00 - NGrams - Teradata Vantage

Teradata® Vantage Machine Learning Engine Analytic Function Reference

Product
Teradata Vantage
Release Number
1.0
8.00
Release Date
May 2019
Content Type
Programming Reference
Publication ID
B700-4003-098K
Language
English (United States)

The NGrams function tokenizes (splits) an input stream of text and outputs n multigrams (called n -grams) based on the specified delimiter and reset parameters. NGrams provides more flexibility than standard tokenization when performing text analysis. Many two-word phrases carry important meaning (for example, "machine learning") that unigrams (single-word tokens) do not capture. This, combined with additional analytical techniques, can be useful for performing sentiment analysis, topic identification, and document classification.

NGrams considers each input row to be one document, and returns a row for each unique n-gram in each document. NGrams also returns, for each document, the counts of each n-gram and the total number of n-grams.

For general information about tokenization, see http://en.wikipedia.org/wiki/Lexical_analysis#Tokenizer.