Input table: stopwords
word ---- a an the
Input table: test_table
id paragraph -- ---------------------------------------------- 1 Programmers program with programming languages 2 The quick brown fox jumps over the lazy dog
SQL Call
SELECT * FROM TD_TextParser (
ON test_table AS InputTable
ON stopwords As StopWordsTable DIMENSION
USING
TextColumn ('paragraph')
StemTokens ('true')
RemoveStopWords ('true')
Accumulate ('id')
) as dt ORDER BY id,token;
The query performs the following operations:
- Removes the stopwords from the text in the Paragraph column
- Splits the text in the Paragraph column and creates a row for each word in the output table
- Copies the ID column from the input table to the output table
Output Table
id token -- -------- 1 languag 1 program 1 program 1 programm 1 with 2 brown 2 dog 2 fox 2 jump 2 lazi 2 over 2 quick