TextParser Example: StopWordsList, No StemExceptions - Teradata Vantage

Machine Learning Engine Analytic Function Reference

Product
Teradata Vantage
Release Number
8.10
1.1
Published
October 2019
Language
English (United States)
Last Update
2019-12-31
dita:mapPath
ima1540829771750.ditamap
dita:ditavalPath
jsj1481748799576.ditaval
dita:id
B700-4003
lifecycle
previous
Product Category
Teradata Vantageā„¢

Input

  • InputTable: complaints, a log of vehicle complaints.

    The category column indicates whether the vehicle was in a crash.

  • Stop words file: stopwords.txt, which is preinstalled on ML Engine (shown in TextClassifierTrainer Example)
complaints
doc_id text_data category
1 consumer was driving approximately 45 mph hit a deer with the front bumper and then ran into an embankment head-on passenger's side air bag did deploy hit windshield and deployed outward. driver's side airbag cover opened but did not inflate it was still folded causing injuries. crash
2 when vehicle was involved in a crash totalling vehicle driver's side/ passenger's side air bags did not deploy. vehicle was making a left turn and was hit by a ford f350 traveling about 35 mph on the front passenger's side. driver hit his head-on the steering wheel. hurt his knee and received neck and back injuries. crash
3 consumer has experienced following problems; 1.) both lower ball joints wear out excessively; 2.) head gasket leaks; and 3.) cruise control would shut itself off while driving without foot pressing on brake pedal. no_crash
... ... ...

SQL Call

SELECT * FROM TextParser (
  ON complaints
  USING
  TextColumn ('text_data')
  ConvertToLowerCase ('true')
  StemTokens ('false')
  OutputByWord ('true')
  Punctuation ('\[.,?\!\]')
  RemoveStopWords ('true')
  ListPositions ('true')
  Accumulate ('doc_id', 'category')
  StopWordsList ('stopwords.txt')
) AS dt ORDER BY doc_id,category,token,frequency,location;

Output

 doc_id category token          frequency location   
 ------ -------- -------------- --------- ---------- 
      1 crash    45                     1 4         
      1 crash    air                    1 22        
      1 crash    airbag                 1 33        
      1 crash    approximately          1 3         
      1 crash    bag                    1 23        
      1 crash    bumper                 1 12        
      1 crash    causing                1 44        
      1 crash    consumer               1 0         
      1 crash    cover                  1 34        
      1 crash    deer                   1 8         
      1 crash    deploy                 1 25        
      1 crash    deployed               1 29        
      1 crash    did                    2 24,37     
      1 crash    driver's               1 31        
      1 crash    driving                1 2         
      1 crash    embankment             1 18        
      1 crash    folded                 1 43        
      1 crash    front                  1 11        
      1 crash    head-on                1 19        
      1 crash    hit                    2 6,26      
      1 crash    inflate                1 39        
      1 crash    injuries               1 45        
      1 crash    it                     1 40        
      1 crash    mph                    1 5         
      1 crash    not                    1 38        
      1 crash    opened                 1 35        
      1 crash    outward                1 30        
      1 crash    passenger's            1 20        
      1 crash    ran                    1 15        
      1 crash    side                   2 21,32     
      1 crash    still                  1 42        
      1 crash    then                   1 14        
      1 crash    windshield             1 27        
      2 crash    35                     1 33        
      2 crash    about                  1 32        
      2 crash    air                    1 13        
      2 crash    back                   1 54        
      2 crash    bags                   1 14        
      2 crash    by                     1 27        
      2 crash    crash                  1 6         
      2 crash    deploy                 1 17        
      2 crash    did                    1 15        
      2 crash    driver                 1 40        
      2 crash    driver's               1 9         
      2 crash    f350                   1 30        
      2 crash    ford                   1 29        
      2 crash    front                  1 37        
      2 crash    head-on                1 43        
      2 crash    his                    2 42,48     
      2 crash    hit                    2 26,41     
      2 crash    hurt                   1 47        
      2 crash    injuries               1 55        
      2 crash    involved               1 3         
      2 crash    knee                   1 49        
      2 crash    left                   1 22        
      2 crash    making                 1 20        
      2 crash    mph                    1 34        
      2 crash    neck                   1 52        
      2 crash    not                    1 16        
      2 crash    on                     1 35        
      2 crash    passenger's            2 11,38     
      2 crash    received               1 51        
      2 crash    side                   2 12,39     
      2 crash    side/                  1 10        
      2 crash    steering               1 45        
      2 crash    totalling              1 7         
      2 crash    traveling              1 31        
      2 crash    turn                   1 23        
      2 crash    vehicle                3 1,8,18    
      2 crash    wheel                  1 46        
      2 crash    when                   1 0         
      3 no_crash 1)                     1 5         
      3 no_crash 2)                     1 13        
      3 no_crash 3)                     1 18        
      3 no_crash ball                   1 8         
      3 no_crash both                   1 6         
      3 no_crash brake                  1 31        
      3 no_crash consumer               1 0         
      3 no_crash control                1 20        
      3 no_crash cruise                 1 19        
      3 no_crash driving                1 26        
      3 no_crash excessively;           1 12        
      3 no_crash experienced            1 2         
      3 no_crash following              1 3         
      3 no_crash foot                   1 28        
      3 no_crash gasket                 1 15        
      3 no_crash has                    1 1         
      3 no_crash head                   1 14        
      3 no_crash itself                 1 23        
      3 no_crash joints                 1 9         
      3 no_crash leaks;                 1 16        
      3 no_crash lower                  1 7         
      3 no_crash off                    1 24        
      3 no_crash on                     1 30        
      3 no_crash out                    1 11        
      3 no_crash pedal                  1 32        
      3 no_crash pressing               1 29        
      3 no_crash problems;              1 4         
      3 no_crash shut                   1 22        
      3 no_crash wear                   1 10        
      3 no_crash while                  1 25        
      3 no_crash without                1 27        
      3 no_crash would                  1 21        
      4 no_crash after                  1 6         
      4 no_crash back                   1 18        
      4 no_crash been                   1 40        
      4 no_crash case                   2 1,36      
      4 no_crash completed              1 10        
      4 no_crash consumer               1 15        
      4 no_crash dealer                 2 20,22     
      4 no_crash driveshaft             1 31        
      4 no_crash has                    1 39        
      4 no_crash heard                  1 13        
      4 no_crash hitting                1 33        
      4 no_crash informed               1 26        
      4 no_crash intermittently         1 14        
      4 no_crash manufacturer           1 38        
      4 no_crash noise                  1 11        
      4 no_crash notfied                1 41        
      4 no_crash owner                  1 28        
      4 no_crash recall                 1 5         
      4 no_crash reinspected            1 23        
      4 no_crash repaired               1 3         
      4 no_crash that                   1 29        
      4 no_crash took                   1 16        
      4 no_crash transfer               2 0,35      
      4 no_crash under                  1 4         
      4 no_crash vehicle                2 17,24     
      4 no_crash work                   1 8         
      5 no_crash &                      2 21,27     
      5 no_crash 10mph                  1 8         
      5 no_crash 3                      1 14        
      5 no_crash accurate               1 41        
      5 no_crash almost                 1 33        
      5 no_crash also                   1 36        
      5 no_crash at                     1 19        
      5 no_crash be                     1 12        
      5 no_crash blew                   1 34        
      5 no_crash by                     1 56        
      5 no_crash checked                1 18        
      5 no_crash dealership             1 20        
      5 no_crash defect                 1 32        
      5 no_crash does                   1 38        
      5 no_crash factory                1 31        
      5 no_crash fail                   1 48        
      5 no_crash had                    1 16        
      5 no_crash if                     1 43        
      5 no_crash increasedit            1 46        
      5 no_crash informed               1 23        
      5 no_crash it's                   1 29        
      5 no_crash just                   1 7         
      5 no_crash keep                   1 40        
      5 no_crash manufacturer           1 57        
      5 no_crash mechanic               1 55        
      5 no_crash not                    1 39        
      5 no_crash over                   1 13        
      5 no_crash referred               1 53        
      5 no_crash rpms                   1 10        
      5 no_crash slip                   1 4         
      5 no_crash speed                  1 44        
      5 no_crash speedometer            1 37        
      5 no_crash speeds                 1 42        
      5 no_crash start                  1 2         
      5 no_crash stuck                  1 26        
      5 no_crash that                   1 28        
      5 no_crash thousand               1 15        
      5 no_crash transmission           2 0,24      
      5 no_crash traveling              1 6         
      5 no_crash up                     1 35        
      5 no_crash vehicle                1 17        
      5 no_crash when                   1 5         
      5 no_crash work                   1 50        
      5 no_crash would                  3 1,11,47   
      6 no_crash also                   1 21        
      6 no_crash belts/speed            1 27        
      6 no_crash burned                 1 7         
      6 no_crash cable                  1 5         
      6 no_crash coil                   1 9         
      6 no_crash controlcable           1 28        
      6 no_crash could                  1 15        
      6 no_crash crash                  1 20        
      6 no_crash dealer                 1 22        
      6 no_crash defective              1 3         
      6 no_crash drive                  1 26        
      6 no_crash due                    1 0         
      6 no_crash further                1 36        
      6 no_crash have                   1 16        
      6 no_crash ignition               1 4         
      6 no_crash information            1 37        
      6 no_crash performed              1 30        
      6 no_crash please                 1 34        
      6 no_crash provide                1 35        
      6 no_crash r&r                    1 25        
      6 no_crash replaced               1 23        
      6 no_crash resulted               1 17        
      6 no_crash stalled                1 12        
      6 no_crash tune                   1 32        
      6 no_crash unexpectedly           1 13        
      6 no_crash up                     1 33        
      6 no_crash vehicle                2 11,31     
      6 no_crash which                  2 6,14      
      7 no_crash &                      1 16        
      7 no_crash 97v017000              1 28        
      7 no_crash by                     1 25        
      7 no_crash do                     1 22        
      7 no_crash have                   1 12        
      7 no_crash jiggle                 1 14        
      7 no_crash move                   1 20        
      7 no_crash not                    1 8         
      7 no_crash off/on                 1 24        
      7 no_crash on                     1 4         
      7 no_crash properly               1 10        
      7 no_crash recall                 1 27        
      7 no_crash switch                 2 1,15      
      7 no_crash themselves             1 26        
      7 no_crash then                   1 17        
      7 no_crash turn                   1 23        
      7 no_crash turned                 1 3         
      7 no_crash when                   1 0         
      7 no_crash windshield             1 5         
      7 no_crash wipers                 3 6,18,21   
      7 no_crash work                   1 9         
      7 no_crash would                  3 7,11,19   
      8 no_crash consumer               1 0         
      8 no_crash driving                1 2         
      8 no_crash happened               1 13        
      8 no_crash periodcally            1 14        
      8 no_crash rain                   1 5         
      8 no_crash stopped                1 11        
      8 no_crash storm                  1 6         
      8 no_crash when                   1 7         
      8 no_crash windshield             1 9         
      8 no_crash wipers                 1 10        
      9 no_crash *ml                    1 21        
      9 no_crash 66900                  1 1         
      9 no_crash at                     2 0,16      
      9 no_crash expense                1 18        
      9 no_crash first                  1 11        
      9 no_crash gear                   1 12        
      9 no_crash has                    1 4         
      9 no_crash made                   1 15        
      9 no_crash malfunctioned          1 5         
      9 no_crash miles                  1 2         
      9 no_crash not                    1 8         
      9 no_crash owner's                1 17        
      9 no_crash reimbursement          1 20        
      9 no_crash repairs                1 13        
      9 no_crash shift                  1 9         
      9 no_crash transmission           1 3         
      9 no_crash wants                  1 19        
      9 no_crash were                   1 14        
     10 no_crash 1998                   1 33        
     10 no_crash aware                  1 14        
     10 no_crash been                   1 21        
     10 no_crash by                     1 27        
     10 no_crash corrected              1 22        
     10 no_crash has                    1 19        
     10 no_crash hill                   1 29        
     10 no_crash incline                1 6         
     10 no_crash it                     1 7         
     10 no_crash its                    1 10        
     10 no_crash manufactured           1 31        
     10 no_crash manufacturer           1 12        
     10 no_crash not                    1 20        
     10 no_crash of                     1 15        
     10 no_crash on                     2 4,9       
     10 no_crash own                    1 11        
     10 no_crash owned                  1 26        
     10 no_crash problem                2 17,18     
     10 no_crash recker                 1 30        
     10 no_crash rolled                 1 8         
     10 no_crash sitting                1 3         
     10 no_crash truck                  2 1,24      
     10 no_crash walnut                 1 28        
     10 no_crash when                   1 0         
     11 crash    approximately          1 23        
     11 crash    been                   1 20        
     11 crash    building               1 17        
     11 crash    car                    3 0,7,18    
     11 crash    condition              1 32        
     11 crash    crashed                1 11        
     11 crash    engine                 1 1         
     11 crash    fence                  1 14        
     11 crash    for                    1 29        
     11 crash    forward                1 9         
     11 crash    had                    1 19        
     11 crash    high                   1 30        
     11 crash    idle                   1 31        
     11 crash    incident               1 28        
     11 crash    lurched                1 8         
     11 crash    one                    1 24        
     11 crash    park                   1 6         
     11 crash    prior                  1 26        
     11 crash    raced                  1 2         
     11 crash    shop                   1 22        
     11 crash    slowing                1 4         
     11 crash    week                   1 25        
     11 crash    while                  1 3         
     12 crash    65                     1 5         
     12 crash    70mph                  1 7         
     12 crash    airbags                1 15        
     12 crash    another                1 2         
     12 crash    at                     1 4         
     12 crash    dealer                 1 17        
     12 crash    deployed               1 16        
     12 crash    driver's               1 10        
     12 crash    ended                  1 1         
     12 crash    has                    1 18        
     12 crash    neither                1 9         
     12 crash    or                     1 12        
     12 crash    passenger's            1 13        
     12 crash    rear                   1 0         
     12 crash    side                   2 11,14     
     12 crash    vehicle                2 3,19      
     13 no_crash around                 1 27        
     13 no_crash coming                 1 25        
     13 no_crash compartment            1 17        
     13 no_crash drivers                1 28        
     13 no_crash ea02-025               1 34        
     13 no_crash engine                 1 16        
     13 no_crash fire                   2 8,24      
     13 no_crash for                    1 4         
     13 no_crash from                   1 26        
     13 no_crash front                  1 30        
     13 no_crash hour                   1 6         
     13 no_crash left                   1 12        
     13 no_crash of                     1 14        
     13 no_crash on                     1 10        
     13 no_crash owner                  1 22        
     13 no_crash owners                 1 18        
     13 no_crash parked                 1 3         
     13 no_crash referenced             1 32        
     13 no_crash saw                    1 23        
     13 no_crash side                   2 13,29     
     13 no_crash smelled                1 20        
     13 no_crash smoke                  1 21        
     13 no_crash son                    1 19        
     13 no_crash started                1 9         
     13 no_crash vehicle                1 1         
     13 no_crash wheel                  1 31        
     13 no_crash while                  1 0         
     14 no_crash                        1 14        
     14 no_crash 99v029000              1 6         
     14 no_crash after                  1 0         
     14 no_crash airbag                 1 10        
     14 no_crash been                   1 21        
     14 no_crash dealer                 1 16        
     14 no_crash has                    1 20        
     14 no_crash ignition               1 7         
     14 no_crash light                  1 11        
     14 no_crash manufacturer           1 19        
     14 no_crash notified               1 22        
     14 no_crash on                     1 13        
     14 no_crash recall                 1 5         
     14 no_crash repaired               1 3         
     14 no_crash stayed                 1 12        
     14 no_crash switch                 1 8         
     14 no_crash under                  1 4         
     14 no_crash vehicle                1 1         
     15 no_crash 4                      1 27        
     15 no_crash alternator/            1 20        
     15 no_crash battery                1 21        
     15 no_crash become                 1 13        
     15 no_crash cannot                 1 33        
     15 no_crash causing                2 6,37      
     15 no_crash change                 1 19        
     15 no_crash consumer               1 16        
     15 no_crash control                1 1         
     15 no_crash defect                 1 30        
     15 no_crash determine              1 34        
     15 no_crash electrical             1 0         
     15 no_crash engine                 1 11        
     15 no_crash had                    1 17        
     15 no_crash inoperative            1 15        
     15 no_crash module                 2 2,25      
     15 no_crash occurring              1 32        
     15 no_crash out                    1 5         
     15 no_crash problem                1 39        
     15 no_crash replaced               1 26        
     15 no_crash shortening             1 4         
     15 no_crash stall                  1 10        
     15 no_crash starter                1 23        
     15 no_crash still                  1 31        
     15 no_crash times                  1 28        
     15 no_crash totally                1 14        
     15 no_crash vehicle                1 8         
     15 no_crash what                   1 35        
     16 no_crash 68000                  1 1         
     16 no_crash also                   1 17        
     16 no_crash at                     1 0         
     16 no_crash broke                  1 5         
     16 no_crash caused                 1 18        
     16 no_crash causing                1 10        
     16 no_crash down                   1 23        
     16 no_crash housing                1 8         
     16 no_crash loss                   1 12        
     16 no_crash miles                  1 2         
     16 no_crash of                     1 13        
     16 no_crash off                    1 6         
     16 no_crash power                  2 3,14      
     16 no_crash pump                   1 9         
     16 no_crash shut                   1 22        
     16 no_crash steering               2 4,15      
     16 no_crash total                  1 11        
     16 no_crash vehicle                1 20        
     16 no_crash which                  1 16        
     17 crash    50                     1 14        
     17 crash    80                     1 17        
     17 crash    air                    2 25,37     
     17 crash    airbags                1 4         
     17 crash    another                1 10        
     17 crash    approximately          1 13        
     17 crash    at                     2 12,16     
     17 crash    bags                   2 26,38     
     17 crash    consumer               1 8         
     17 crash    deploy                 3 7,29,41   
     17 crash    determine              1 35        
     17 crash    did                    4 5,27,33,39
     17 crash    driver                 1 30        
     17 crash    dual                   1 3         
     17 crash    head-on                1 22        
     17 crash    hit                    1 19        
     17 crash    impact                 1 24        
     17 crash    injuriesdealer         1 32        
     17 crash    mph                    1 18        
     17 crash    mphand                 1 15        
     17 crash    not                    4 6,28,34,40
     17 crash    occasions              1 2         
     17 crash    on                     1 0         
     17 crash    rearended              1 9         
     17 crash    sustained              1 31        
     17 crash    truck                  1 21        
     17 crash    two                    1 1         
     17 crash    upon                   1 23        
     17 crash    vehicle                1 11        
     17 crash    why                    1 36        
     18 no_crash leaking                1 2         
     18 no_crash sunroof                1 0         
     18 no_crash yh                     1 3         
     19 no_crash be                     1 9         
     19 no_crash frame                  1 3         
     19 no_crash from                   1 5         
     19 no_crash manufacturer           1 7         
     19 no_crash motor                  1 0         
     19 no_crash notified               1 10        
     19 no_crash separated              1 4         
     19 no_crash vehicle                1 6         
     20 no_crash about                  1 19        
     20 no_crash bearing                1 3         
     20 no_crash brake's                1 17        
     20 no_crash broke                  1 4         
     20 no_crash can't                  1 25        
     20 no_crash causing                1 5         
     20 no_crash consumer               1 15        
     20 no_crash dealer                 1 24        
     20 no_crash determine              1 26        
     20 no_crash down                   1 14        
     20 no_crash four                   1 20        
     20 no_crash front                  1 1         
     20 no_crash had                    1 16        
     20 no_crash left                   1 11        
     20 no_crash problem                1 28        
     20 no_crash pull                   1 8         
     20 no_crash rear                   1 0         
     20 no_crash replaced               1 18        
     20 no_crash slowing                1 13        
     20 no_crash still                  1 23        
     20 no_crash times                  1 21        
     20 no_crash vehicle                1 6         
     20 no_crash wheel                  1 2         
     20 no_crash when                   1 12

Download a zip file of all examples and a SQL script file that creates their input tables from the attachment in the left sidebar.