TD_TextParser Example | TextParser | Teradata Vantage - Example 1: Using TD_TextParser with StopWords Table - Analytics Database

Database Analytic Functions

Deployment
VantageCloud
VantageCore
Edition
Enterprise
IntelliFlex
VMware
Product
Analytics Database
Release Number
17.20
Published
June 2022
Language
English (United States)
Last Update
2024-10-04
dita:mapPath
gjn1627595495337.ditamap
dita:ditavalPath
ayr1485454803741.ditaval
dita:id
jmh1512506877710
lifecycle
latest
Product Category
Teradata Vantageā„¢

Input table: stopwords

word 
---- 
a   
an  
the 

Input table: test_table

id paragraph                                      
-- ---------------------------------------------- 
 1 Programmers program with programming languages
 2 The quick brown fox jumps over the lazy dog   

SQL Call

SELECT * FROM TD_TextParser (
ON test_table AS InputTable
ON stopwords As StopWordsTable DIMENSION
USING
TextColumn ('paragraph')
StemTokens ('true')
RemoveStopWords ('true')
Accumulate ('id')
) as dt ORDER BY id,token;
The query performs the following operations:
  • Removes the stopwords from the text in the Paragraph column
  • Splits the text in the Paragraph column and creates a row for each word in the output table
  • Copies the ID column from the input table to the output table

Output Table

 id token    
 -- -------- 
  1 languag 
  1 program 
  1 program 
  1 programm
  1 with    
  2 brown   
  2 dog     
  2 fox     
  2 jump    
  2 lazi    
  2 over    
  2 quick