Teradata Package for R Function Reference | 17.20 - NGramSplitter - Teradata Package for R - Look here for syntax, methods and examples for the functions included in the Teradata Package for R.

Teradata® Package for R Function Reference

Deployment
VantageCloud
VantageCore
Edition
Enterprise
IntelliFlex
VMware
Product
Teradata Package for R
Release Number
17.20
Published
March 2024
Language
English (United States)
Last Update
2024-05-03
dita:id
TeradataR_FxRef_Enterprise_1720
Product Category
Teradata Vantage

NGramSplitter

Description

The td_ngramsplitter_sqle function tokenizes (splits) an input stream of text and outputs n multigrams (called n-grams) based on the specified delimiter and reset parameters. td_ngramsplitter_sqle provides more flexibility than standard tokenization when performing text analysis. Many two-word phrases carry important meaning (for example, "machine learning") that unigrams (single-word tokens) do not capture. This, combined with additional analytical techniques, can be useful for performing sentiment analysis, topic identification and document classification.

Note: This function is only available when tdplyr is connected to Vantage 1.1 or later versions.

Usage

  td_ngramsplitter_sqle (
      data = NULL,
      text.column = NULL,
      delimiter = " ",
      grams = NULL,
      overlapping = TRUE,
      to.lower.case = TRUE,
      punctuation = "`~#^&*()-",
      reset = ".,?!",
      total.gram.count = FALSE,
      total.count.column = "totalcnt",
      accumulate = NULL,
      n.gram.column = "ngram",
      num.grams.column = "n",
      frequency.column = "frequency",
      ...
  )

Arguments

data

Required Argument.
Specifies input tbl_teradata, where each row contains a document
to be tokenized. The input tbl_teradata can have additional rows, some or all of which the function returns in the output table.
Types: tbl_teradata

text.column

Required Argument.
Specifies the name of the column that contains the input text. The column must have a SQL string data type.
Types: character

delimiter

Optional Argument.
Specifies a character or string that separates words in the input text. The default value is the set of all whitespace characters which includes the characters for space, tab, newline, carriage return and some others.
Default Value: " "(space)
Types: character

grams

Required Argument.
Specifies the length, in words, of each n-gram (that is, the value of n).
A value_range has the syntax integer1 - integer2, where integer1 <= integer2.
The values of n, integer1, and integer2 must be positive.
Types: character

overlapping

Optional Argument.
Specifies a Boolean value that specifies whether the function allows overlapping n-grams. When this value is "true" (the default), each word in each sentence starts an n-gram, if enough words follow it (in the same sentence) to form a whole n-gram of the specified size. For information on sentences, see the description of the reset argument.
Default Value: TRUE
Types: logical

to.lower.case

Optional Argument.
Specifies a Boolean value that specifies whether the function converts all letters in the input text to lowercase.
Default Value: TRUE
Types: logical

punctuation

Optional Argument.
Specifies a string that specifies the punctuation characters for the function to remove before evaluating the input text.
Default Value: "'~#^&*()-"
Types: character

reset

Optional Argument.
Specifies a string that specifies the character or string that ends a sentence.
At the end of a sentence, the function discards any partial n-grams and searches for the next n-gram at the beginning of the next sentence. An n-gram cannot span two sentences.
Default Value: ".,?!"
Types: character

total.gram.count

Optional Argument.
Specifies whether the function returns the total
number of ngrams in the document (that is, in the row). If you specify "true", then the name of the returned column is specified by the total.count.column argument.
Note: The total number of n-grams is not necessarily the number of unique ngrams.
Default Value: FALSE
Types: logical

total.count.column

Optional Argument.
Specifies the name of the column to return if the value of the total.gram.count argument is TRUE.
Default Value: "totalcnt"
Types: character

accumulate

Optional Argument.
Specifies the names of the columns to return for each n-gram. These columns cannot have the same names as those specified by the arguments ngram, num.grams.column, and total.count.column. By default, the function returns all input columns for each n-gram.
Types: character OR vector of Strings (character)

n.gram.column

Optional Argument.
Specifies the name of the column that is to contain the generated n-grams.
Default Value: "ngram"
Types: character

num.grams.column

Optional Argument.
Specifies the name of the column that is to contain the length of n-gram (in words).
Default Value: "n"
Types: character

frequency.column

Optional Argument.
Specifies the name of the column that is to contain the count of each unique n-gram (that is, the number of times that each unique n-gram appears in the document).
Default Value: "frequency"
Types: character

...

Specifies the generic keyword arguments SQLE functions accept.
Below are the generic keyword arguments:

persist:
Optional Argument.
Specifies whether to persist the results of the function in a table or not.
When set to TRUE, results are persisted in a table; otherwise, results are garbage collected at the end of the session.
Default Value: FALSE
Types: logical

volatile:
Optional Argument.
Specifies whether to put the results of the function in a volatile table or not.
When set to TRUE, results are stored in a volatile table, otherwise not.
Default Value: FALSE
Types: logical

Function allows the user to partition, hash, order or local order the input data. These generic arguments are available for each argument that accepts tbl_teradata as input and can be accessed as:

  • "<input.data.arg.name>.partition.column" accepts character OR vector of Strings (character) (Strings)

  • "<input.data.arg.name>.hash.column" accepts character OR vector of Strings (character) (Strings)

  • "<input.data.arg.name>.order.column" accepts character OR vector of Strings (character) (Strings)

  • "local.order.<input.data.arg.name>" accepts logical

Note:
These generic arguments are supported by tdplyr if the underlying SQL Engine function supports, else an exception is raised.

Value

Function returns an object of class "td_ngramsplitter_sqle" which is a named list containing object of class "tbl_teradata".
Named list member(s) can be referenced directly with the "$" operator using the name(s):result

Examples

  
    
    # Get the current context/connection.
    con <- td_get_context()$connection
    
    # Load the example data.
    loadExampleData("ngram_example", "paragraphs_input")
    
    # Create tbl_teradata object.
    paragraphs_input <- tbl(con, "paragraphs_input")
    
    # Check the list of available analytic functions.
    display_analytic_functions()
    
    # Example 1: Creating tbl_teradata by calculating the
    #            similarity between two strings.
    obj <- td_ngramsplitter_sqle(data=paragraphs_input,
                                 text.column='paratext',
                                 n.gram.column='ngram',
                                 num.grams.column='n',
                                 frequency.column='frequency',
                                 total.count.column='totalcnt',
                                 grams='4-6',
                                 overlapping=TRUE,
                                 to.lower.case=TRUE,
                                 delimiter=' ',
                                 punctuation='`~#^&*()-',
                                 reset='.,?!',
                                 total.gram.count=FALSE)
    
    # Print the result.
    print(obj$result)