Input table: stopwords
word ---- a an the
Input table: test_table
id paragraph -- ---------------------------------------------- 1 Programmers program with programming languages 2 The quick brown fox jumps over the lazy dog
SQL Call
SELECT * FROM TD_TextParser ( ON test_table AS InputTable ON stopwords As StopWordsTable DIMENSION USING TextColumn ('paragraph') StemTokens ('true') RemoveStopWords ('true') Accumulate ('id') ) as dt ORDER BY id,token;
The query performs the following operations:
- Removes the stopwords from the text in the Paragraph column
- Splits the text in the Paragraph column and creates a row for each word in the output table
- Copies the ID column from the input table to the output table
Output Table
id token -- -------- 1 languag 1 program 1 program 1 programm 1 with 2 brown 2 dog 2 fox 2 jump 2 lazi 2 over 2 quick