1.1 - 8.10 - TextParser Example: StemExceptions, No StopWordsList - Teradata Vantage

Teradata Vantage™ - Machine Learning Engine Analytic Function Reference

Product
Teradata Vantage
Release Number
1.1
8.10
Release Date
October 2019
Content Type
Programming Reference
Publication ID
B700-4003-079K
Language
English (United States)

Input

complaints_mini
doc_id text_data category
1 consumer was driving approximately 45 mph hit a deer with the front bumper and then ran into an enbankment head-on passenger's side air bag did deploy hit windshield and deployed outward. driver's side airbag cover opened but did not inflate it was still folded causing injuries. crash
2 when vehicle was involved in a crash totalling vehicle driver's side/ passenger's side air bags did not deploy. vehicle was making a left turn and was hit by a ford f350 traveling about 35 mph on the front passenger's side. driver hit his head-on the steering wheel. hurt his knee and received neck and back injuries. crash

The stemming exceptions table, stemmingexception.text, contains:

consumer customer
enbankment embankment

SQL Call

SELECT * FROM TextParser (
  ON complaints_mini
  USING
  TextColumn ('text_data')
  ConvertToLowerCase ('true')
  StemTokens ('true')
  OutputByWord ('false')
  Punctuation ('\[.,?\!\]')
  Accumulate ('doc_id', 'category')
  StemExceptions ('stemmingexception.txt')
) AS dt ORDER BY doc_id;

Output

 doc_id category tokens                                                                                                                                                                                                                                                                                     
 ------ -------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 
      1 crash    customer was drive approxim 45 mph hit a deer with the front bumper and then ran into an embankment head-on passeng side air bag did deploy hit windshield and deploy outward driver side airbag cover open but did not inflat it was still fold caus injuri                              
      2 crash    when vehicl was involv in a crash total vehicl driver side/ passeng side air bag did not deploy vehicl was make a left turn and was hit by a ford f350 travel about 35 mph on the front passeng side driver hit his head-on the steer wheel hurt his knee and receiv neck and back injuri

Download a zip file of all examples and a SQL script file that creates their input tables from the attachment in the left sidebar.