TextParser Example: StopWordsTable, No StemTokens | Teradata Vantage - TextParser Example: StopWordsTable, No StemTokens - Teradata Vantage

Machine Learning Engine Analytic Function Reference

Product
Teradata Vantage
Release Number
9.02
9.01
2.0
1.3
Published
February 2022
Language
English (United States)
Last Update
2022-02-10
dita:mapPath
rnn1580259159235.ditamap
dita:ditavalPath
ybt1582220416951.ditaval
dita:id
B700-4003
lifecycle
previous
Product Category
Teradata Vantageā„¢

Input

SQL Call

SELECT * FROM TextParser (
  ON complaints PARTITION BY ANY
  ON stopwords1_table as StopWordsTable DIMENSION
  USING TextColumn ('text_data')
  Accumulate ('doc_id','category')
  TokenColName ('token')
  FrequencyColName ('frequency')
  TotalColName ('total_count')
  PositionColName ('position')
  ConvertToLowerCase ('True')
  StemTokens ('False')
  OutputByWord ('True')
  RemoveStopWords ('True')
  OutputTotalWords ('False')
  Punctuation ('[.,!?]')
  ListPositions ('true')
) as dt order by doc_id;

Output

doc_id category token                  frequency position
------ -------- ---------------------- --------- ------------------
     1 crash    Snippet:windshield             1 Snippet:27
     1 crash    Snippet:ran                    1 Snippet:15
    ...
     1 crash    Snippet:hit                    2 Snippet:6,26
     2 crash    Snippet:totalling              1 Snippet:7
     2 crash    Snippet:deploy                 1 Snippet:17
    ...
     2 crash    Snippet:did                    1 Snippet:15
     3 no_crash Snippet:pressing               1 Snippet:29
     3 no_crash Snippet:3)                     1 Snippet:18
    ...
     3 no_crash Snippet:shut                   1 Snippet:22
     4 no_crash Snippet:intermittently         1 Snippet:14
     4 no_crash Snippet:been                   1 Snippet:40
    ...
     4 no_crash Snippet:completed              1 Snippet:10
     5 no_crash Snippet:referred               1 Snippet:53
     5 no_crash Snippet:would                  3 Snippet:1,11,47
    ...
     5 no_crash Snippet:start                  1 Snippet:2
     6 no_crash Snippet:vehicle                2 Snippet:11,31
     6 no_crash Snippet:ignition               1 Snippet:4
    ...
     6 no_crash Snippet:unexpectedly           1 Snippet:13
     7 no_crash Snippet:turn                   1 Snippet:23
     7 no_crash Snippet:themselves             1 Snippet:26
    ...
     7 no_crash Snippet:by                     1 Snippet:25
     8 no_crash Snippet:storm                  1 Snippet:6
     8 no_crash Snippet:driving                1 Snippet:2
    ...
     8 no_crash Snippet:rain                   1 Snippet:5
     9 no_crash Snippet:*ml                    1 Snippet:21
     9 no_crash Snippet:reimbursement          1 Snippet:20
    ...
     9 no_crash Snippet:at                     2 Snippet:0,16
    10 no_crash Snippet:manufacturer           1 Snippet:12
    10 no_crash Snippet:own                    1 Snippet:11
    ...
    10 no_crash Snippet:aware                  1 Snippet:14
    11 crash Snippet:park                      1 Snippet:6
    11 crash Snippet:slowing                   1 Snippet:4
    ...
    11 crash Snippet:lurched                   1 Snippet:8
    12 crash Snippet:or                        1 Snippet:12
    12 crash Snippet:has                       1 Snippet:18
    ...
    12 crash Snippet:70mph                     1 Snippet:7
    13 no_crash Snippet:while                  1 Snippet:0
    13 no_crash Snippet:ea02-025               1 Snippet:34
    ...
    13 no_crash Snippet:vehicle                1 Snippet:1
    14 no_crash Snippet:notified               1 Snippet:22
    14 no_crash Snippet:been                   1 Snippet:21
    ...
    14 no_crash Snippet:after                  1 Snippet:0
    15 no_crash Snippet:still                  1 Snippet:31
    15 no_crash Snippet:defect                 1 Snippet:30
    ...
    16 no_crash Snippet:down                   1 Snippet:23
    16 no_crash Snippet:shut                   1 Snippet:22
    ...
    16 no_crash Snippet:at                     1 Snippet:0
    17 crash Snippet:not                       4 Snippet:6,28,34,40
    17 crash Snippet:occasions                 1 Snippet:2
    ...
    17 crash Snippet:dual                      1 Snippet:3
    18 no_crash Snippet:yh                     1 Snippet:3
    18 no_crash Snippet:leaking                1 Snippet:2
    18 no_crash Snippet:sunroof                1 Snippet:0
    19 no_crash Snippet:be                     1 Snippet:9
    19 no_crash Snippet:manufacturer           1 Snippet:7
    ...
    19 no_crash Snippet:motor                  1 Snippet:0
    20 no_crash Snippet:broke                  1 Snippet:4
    20 no_crash Snippet:bearing                1 Snippet:3
    ...
    20 no_crash Snippet:causing                1 Snippet:5