Teradata Package for R Function Reference | 17.20 - TextParser - Teradata Package for R - Look here for syntax, methods and examples for the functions included in the Teradata Package for R.

Teradata® Package for R Function Reference

Deployment
VantageCloud
VantageCore
Edition
Enterprise
IntelliFlex
VMware
Product
Teradata Package for R
Release Number
17.20
Published
March 2024
ft:locale
en-US
ft:lastEdition
2024-05-03
dita:id
TeradataR_FxRef_Enterprise_1720
Product Category
Teradata Vantage

TextParser

Description

The td_text_parser_sqle() function can parse text and perform the following operation:

  • Tokenize the text in the specified column

  • Remove the punctuations from the text and convert the text to lowercase

  • Remove stop words from the text and convert the text to their root forms

  • Create a row for each word in the output tbl_teradata

  • Perform stemming; that is, the function identifies the common root form of a word by removing or replacing word suffixes

Notes:

  • The stems resulting from stemming may not be actual words. For example, the stem for 'communicate' is 'commun' and the stem for 'early' is 'earli' (trailing 'y' is replaced by 'i').

  • This function requires the UTF8 client character set.

  • This function does not support Pass Through Characters (PTCs).

  • For information about PTCs, see Teradata Vantage™ - Analytics Database International Character Set Support.

  • This function does not support KanjiSJIS or Graphic data types.

Usage

  td_text_parser_sqle (
      data = NULL,
      object = NULL,
      text.column = NULL,
      covert.to.lowercase = TRUE,
      stem.tokens = FALSE,
      remove.stopwords = FALSE,
      accumulate = NULL,
      delimiter = " ",
      punctuation = "!#$%&()*+,-./:;?@^_`{|}~",
      token.col.name = NULL,
      ...
  )

Arguments

data

Required Argument.
Specifies the input tbl_teradata.
Types: tbl_teradata

object

Optional Argument.
Specifies the tbl_teradata containing stop words.
Types: tbl_teradata

text.column

Required Argument.
Specifies the name of the input data column whose contents are to be tokenized.
Types: character

covert.to.lowercase

Optional Argument.
Specifies whether to convert the text in "text.column" to lowercase.
Default Value: TRUE
Types: logical

stem.tokens

Optional Argument.
Specifies whether to convert the text in "text.column" to their root forms.
Default Value: FALSE
Types: logical

remove.stopwords

Optional Argument.
Specifies whether to remove stop words from the text in "text.column" before parsing.
Default Value: FALSE
Types: logical

accumulate

Optional Argument.
Specifies the name(s) of input tbl_teradata column(s) to copy to the output. By default, the function copies no input tbl_teradata columns to the output.
Types: character OR vector of Strings (character)

delimiter

Optional Argument.
Specifies the word delimiter to apply to the text in the specified column in the "text.column" element.
Default Value: " "
Types: character

punctuation

Optional Argument.
Specifies the punctuation characters to replace with a space in the input text.
Default Value: "!#$%&()*+,-./:;?@^_'|~"
Types: character

token.col.name

Optional Argument.
Specifies the name for the output column that contains the individual words from the text of the specified column in the "text.column" element.
Types: character

...

Specifies the generic keyword arguments SQLE functions accept. Below
are the generic keyword arguments:

persist:
Optional Argument.
Specifies whether to persist the results of the
function in a table or not. When set to TRUE, results are persisted in a table; otherwise, results are garbage collected at the end of the session.
Default Value: FALSE
Types: logical

volatile:
Optional Argument.
Specifies whether to put the results of the
function in a volatile table or not. When set to TRUE, results are stored in a volatile table, otherwise not.
Default Value: FALSE
Types: logical

Function allows the user to partition, hash, order or local order the input data. These generic arguments are available for each argument that accepts tbl_teradata as input and can be accessed as:

  • "<input.data.arg.name>.partition.column" accepts character or vector of character (Strings)

  • "<input.data.arg.name>.hash.column" accepts character or vector of character (Strings)

  • "<input.data.arg.name>.order.column" accepts character or vector of character (Strings)

  • "local.order.<input.data.arg.name>" accepts logical

Note:
These generic arguments are supported by tdplyr if the underlying SQL Engine function supports, else an exception is raised.

Value

Function returns an object of class "td_text_parser_sqle" which is a named list containing object of class "tbl_teradata".
Named list member(s) can be referenced directly with the "$" operator using the name(s):result

Examples

  
    
    # Get the current context/connection.
    con <- td_get_context()$connection
    
    # Load the example data.
    loadExampleData("textparser_example", "complaints", "stop_words")
    
    # Create tbl_teradata object.
    complaints <- tbl(con, "complaints")
    stop_words <- tbl(con, "stop_words")
    
    # Check the list of available analytic functions.
    display_analytic_functions()
    
    # Example 1 : Remove all the stop words from "text_data" column
    #             and accumulate it by "doc_id" column.
    TextParser_out <- td_text_parser_sqle(data=complaints,
                                          text.column="text_data",
                                          object=stop_words,
                                          remove.stopwords=TRUE,
                                          accumulate="doc_id")
    
    # Print the result.
    print(TextParser_out$result)
    
    # Example 2 : Convert words in "text_data" column into their root forms.
    TextParser_out <- td_text_parser_sqle(data=complaints,
                                          text.column="text_data",
                                          covert.to.lowercase=TRUE,
                                          stem.tokens=TRUE)
    
    # Print the result.
    print(TextParser_out$result)