TextParser
Description
The td_text_parser_sqle()
function can parse text and perform the following operation:
Tokenize the text in the specified column
Remove the punctuations from the text and convert the text to lowercase
Remove stop words from the text and convert the text to their root forms
Create a row for each word in the output tbl_teradata
Perform stemming; that is, the function identifies the common root form of a word by removing or replacing word suffixes
Notes:
The stems resulting from stemming may not be actual words. For example, the stem for 'communicate' is 'commun' and the stem for 'early' is 'earli' (trailing 'y' is replaced by 'i').
This function requires the UTF8 client character set.
This function does not support Pass Through Characters (PTCs).
For information about PTCs, see Teradata Vantage™ - Analytics Database International Character Set Support.
This function does not support KanjiSJIS or Graphic data types.
Usage
td_text_parser_sqle (
data = NULL,
object = NULL,
text.column = NULL,
covert.to.lowercase = TRUE,
stem.tokens = FALSE,
remove.stopwords = FALSE,
accumulate = NULL,
delimiter = " ",
punctuation = "!#$%&()*+,-./:;?@^_`{|}~",
token.col.name = NULL,
...
)
Arguments
data |
Required Argument. |
object |
Optional Argument. |
text.column |
Required Argument. |
covert.to.lowercase |
Optional Argument. |
stem.tokens |
Optional Argument. |
remove.stopwords |
Optional Argument. |
accumulate |
Optional Argument. |
delimiter |
Optional Argument. |
punctuation |
Optional Argument. |
token.col.name |
Optional Argument. |
... |
Specifies the generic keyword arguments SQLE functions accept. Below volatile: Function allows the user to partition, hash, order or local order the input data. These generic arguments are available for each argument that accepts tbl_teradata as input and can be accessed as:
Note: |
Value
Function returns an object of class "td_text_parser_sqle"
which is a named list containing object of class "tbl_teradata".
Named list member(s) can be referenced directly with the "$" operator
using the name(s):result
Examples
# Get the current context/connection.
con <- td_get_context()$connection
# Load the example data.
loadExampleData("textparser_example", "complaints", "stop_words")
# Create tbl_teradata object.
complaints <- tbl(con, "complaints")
stop_words <- tbl(con, "stop_words")
# Check the list of available analytic functions.
display_analytic_functions()
# Example 1 : Remove all the stop words from "text_data" column
# and accumulate it by "doc_id" column.
TextParser_out <- td_text_parser_sqle(data=complaints,
text.column="text_data",
object=stop_words,
remove.stopwords=TRUE,
accumulate="doc_id")
# Print the result.
print(TextParser_out$result)
# Example 2 : Convert words in "text_data" column into their root forms.
TextParser_out <- td_text_parser_sqle(data=complaints,
text.column="text_data",
covert.to.lowercase=TRUE,
stem.tokens=TRUE)
# Print the result.
print(TextParser_out$result)