1.1 - 8.10 - TextTokenizer Example: English Tokenization - Teradata Vantage

Teradata Vantage™ - Machine Learning Engine Analytic Function Reference

Product
Teradata Vantage
Release Number
1.1
8.10
Release Date
October 2019
Content Type
Programming Reference
Publication ID
B700-4003-079K
Language
English (United States)

Input

The InputTable is log of vehicle complaints. The category column indicates whether the car has been in a crash.

complaints
doc_id text_data category
1 consumer was driving approximately 45 mph hit a deer with the front bumper and then ran into an embankment head-on passenger's side air bag did deploy hit windshield and deployed outward. driver's side airbag cover opened but did not inflate it was still folded causing injuries. crash
2 when vehicle was involved in a crash totalling vehicle driver's side/ passenger's side air bags did not deploy. vehicle was making a left turn and was hit by a ford f350 traveling about 35 mph on the front passenger's side. driver hit his head-on the steering wheel. hurt his knee and received neck and back injuries. crash
3 consumer has experienced following problems; 1.) both lower ball joints wear out excessively; 2.) head gasket leaks; and 3.) cruise control would shut itself off while driving without foot pressing on brake pedal. no_crash
... ... ...

SQL Call

SELECT * FROM TextTokenizer (
  ON complaints PARTITION BY ANY
  USING
  InputLanguage ('en')
  OutputDelimiter (' ')
  OutputByWord ('true')
  Accumulate ('doc_id')
  TextColumn ('text_data')
) AS dt ORDER BY doc_id, sn, token;

Output

 doc_id sn token           
 ------ -- --------------- 
      1  1 consumer       
      1  2 was            
      1  3 driving        
      1  4 approximately  
      1  5 45             
      1  6 mph            
      1  7 hit            
      1  8 a              
      1  9 deer           
      1 10 with           
      1 11 the            
      1 12 front          
      1 13 bumper         
      1 14 and            
      1 15 then           
      1 16 ran            
      1 17 into           
      1 18 an             
      1 19 embankment     
      1 20 head-on        
      1 21 passenger's    
      1 22 side           
      1 23 air            
      1 24 bag            
      1 25 did            
      1 26 deploy         
      1 27 hit            
      1 28 windshield     
      1 29 and            
      1 30 deployed       
      1 31 outward        
      1 32 .              
      1 33 driver's       
      1 34 side           
      1 35 airbag         
      1 36 cover          
      1 37 opened         
      1 38 but            
      1 39 did            
      1 40 not            
      1 41 inflate        
      1 42 it             
      1 43 was            
      1 44 still          
      1 45 folded         
      1 46 causing        
      1 47 injuries       
      1 48 .              
      2  1 when           
      2  2 vehicle        
      2  3 was            
      2  4 involved       
      2  5 in             
      2  6 a              
      2  7 crash          
      2  8 totalling      
      2  9 vehicle        
      2 10 driver's       
      2 11 side/          
      2 12 passenger's    
      2 13 side           
      2 14 air            
      2 15 bags           
      2 16 did            
      2 17 not            
      2 18 deploy         
      2 19 .              
      2 20 vehicle        
      2 21 was            
      2 22 making         
      2 23 a              
      2 24 left           
      2 25 turn           
      2 26 and            
      2 27 was            
      2 28 hit            
      2 29 by             
      2 30 a              
      2 31 ford           
      2 32 f350           
      2 33 traveling      
      2 34 about          
      2 35 35             
      2 36 mph            
      2 37 on             
      2 38 the            
      2 39 front          
      2 40 passenger's    
      2 41 side           
      2 42 .              
      2 43 driver         
      2 44 hit            
      2 45 his            
      2 46 head-on        
      2 47 the            
      2 48 steering       
      2 49 wheel          
      2 50 .              
      2 51 hurt           
      2 52 his            
      2 53 knee           
      2 54 and            
      2 55 received       
      2 56 neck           
      2 57 and            
      2 58 back           
      2 59 injuries       
      2 60 .              
      3  1 consumer       
      3  2 has            
      3  3 experienced    
      3  4 following      
      3  5 problems       
      3  6 ;              
      3  7 1              
      3  8 .              
      3  9 )              
      3 10 both           
      3 11 lower          
      3 12 ball           
      3 13 joints         
      3 14 wear           
      3 15 out            
      3 16 excessively    
      3 17 ;              
      3 18 2              
      3 19 .              
      3 20 )              
      3 21 head           
      3 22 gasket         
      3 23 leaks          
      3 24 ;              
      3 25 and            
      3 26 3              
      3 27 .              
      3 28 )              
      3 29 cruise         
      3 30 control        
      3 31 would          
      3 32 shut           
      3 33 itself         
      3 34 off            
      3 35 while          
      3 36 driving        
      3 37 without        
      3 38 foot           
      3 39 pressing       
      3 40 on             
      3 41 brake          
      3 42 pedal          
      3 43 .              
      4  1 transfer       
      4  2 case           
      4  3 was            
      4  4 repaired       
      4  5 under          
      4  6 recall         
      4  7 .              
      4  8 after          
      4  9 the            
      4 10 work           
      4 11 was            
      4 12 completed      
      4 13 noise          
      4 14 was            
      4 15 heard          
      4 16 intermittently 
      4 17 .              
      4 18 consumer       
      4 19 took           
      4 20 vehicle        
      4 21 back           
      4 22 to             
      4 23 dealer         
      4 24 .              
      4 25 the            
      4 26 dealer         
      4 27 reinspected    
      4 28 vehicle        
      4 29 and            
      4 30 informed       
      4 31 the            
      4 32 owner          
      4 33 that           
      4 34 the            
      4 35 driveshaft     
      4 36 was            
      4 37 hitting        
      4 38 the            
      4 39 transfer       
      4 40 case           
      4 41 .              
      4 42 the            
      4 43 manufacturer   
      4 44 has            
      4 45 been           
      4 46 notfied        
      4 47 .              
      5  1 transmission   
      5  2 would          
      5  3 start          
      5  4 to             
      5  5 slip           
      5  6 when           
      5  7 traveling      
      5  8 just           
      5  9 10mph          
      5 10 .              
      5 11 the            
      5 12 rpms           
      5 13 would          
      5 14 be             
      5 15 over           
      5 16 3              
      5 17 thousand       
      5 18 .              
      5 19 had            
      5 20 vehicle        
      5 21 checked        
      5 22 at             
      5 23 dealership     
      5 24 &              
      5 25 was            
      5 26 informed       
      5 27 transmission   
      5 28 was            
      5 29 stuck          
      5 30 &              
      5 31 that           
      5 32 it's           
      5 33 a              
      5 34 factory        
      5 35 defect         
      5 36 almost         
      5 37 blew           
      5 38 up             
      5 39 .              
      5 40 also           
      5 41 speedometer    
      5 42 does           
      5 43 not            
      5 44 keep           
      5 45 accurate       
      5 46 speeds         
      5 47 .              
      5 48 if             
      5 49 speed          
      5 50 is             
      5 51 increasedit    
      5 52 would          
      5 53 fail           
      5 54 to             
      5 55 work           
      5 56 .              
      5 57 this           
      5 58 was            
      5 59 referred       
      5 60 to             
      5 61 mechanic       
      5 62 by             
      5 63 manufacturer   
      5 64 .              
      6  1 due            
      6  2 to             
      6  3 the            
      6  4 defective      
      6  5 ignition       
      6  6 cable          
      6  7 which          
      6  8 burned         
      6  9 the            
      6 10 coil           
      6 11 the            
      6 12 vehicle        
      6 13 stalled        
      6 14 unexpectedly   
      6 15 which          
      6 16 could          
      6 17 have           
      6 18 resulted       
      6 19 in             
      6 20 a              
      6 21 crash          
      6 22 .              
      6 23 also           
      6 24 dealer         
      6 25 replaced       
      6 26 the            
      6 27 r&r            
      6 28 drive          
      6 29 belts/speed    
      6 30 controlcable   
      6 31 and            
      6 32 performed      
      6 33 vehicle        
      6 34 tune           
      6 35 up             
      6 36 .              
      6 37 please         
      6 38 provide        
      6 39 further        
      6 40 information    
      6 41 .              
      7  1 when           
      7  2 switch         
      7  3 is             
      7  4 turned         
      7  5 on             
      7  6 windshield     
      7  7 wipers         
      7  8 would          
      7  9 not            
      7 10 work           
      7 11 properly       
      7 12 .              
      7 13 would          
      7 14 have           
      7 15 to             
      7 16 jiggle         
      7 17 switch         
      7 18 &              
      7 19 then           
      7 20 wipers         
      7 21 would          
      7 22 move           
      7 23 .              
      7 24 wipers         
      7 25 do             
      7 26 turn           
      7 27 off/on         
      7 28 by             
      7 29 themselves     
      7 30 .              
      7 31 recall         
      7 32 97v017000      
      7 33 .              
      8  1 consumer       
      8  2 was            
      8  3 driving        
      8  4 in             
      8  5 a              
      8  6 rain           
      8  7 storm          
      8  8 when           
      8  9 the            
      8 10 windshield     
      8 11 wipers         
      8 12 stopped        
      8 13 this           
      8 14 happened       
      8 15 periodcally    
      8 16 .              
      9  1 at             
      9  2 66900          
      9  3 miles          
      9  4 transmission   
      9  5 has            
      9  6 malfunctioned  
      9  7 and            
      9  8 will           
      9  9 not            
      9 10 shift          
      9 11 into           
      9 12 first          
      9 13 gear           
      9 14 .              
      9 15 repairs        
      9 16 were           
      9 17 made           
      9 18 at             
      9 19 owner's        
      9 20 expense        
      9 21 wants          
      9 22 reimbursement  
      9 23 .              
      9 24 *ml            
     10  1 when           
     10  2 truck          
     10  3 was            
     10  4 sitting        
     10  5 on             
     10  6 an             
     10  7 incline        
     10  8 it             
     10  9 rolled         
     10 10 on             
     10 11 its            
     10 12 own            
     10 13 .              
     10 14 manufacturer   
     10 15 was            
     10 16 aware          
     10 17 of             
     10 18 the            
     10 19 problem        
     10 20 .              
     10 21 problem        
     10 22 has            
     10 23 not            
     10 24 been           
     10 25 corrected      
     10 26 .              
     10 27 the            
     10 28 truck          
     10 29 is             
     10 30 owned          
     10 31 by             
     10 32 walnut         
     10 33 hill           
     10 34 recker         
     10 35 manufactured   
     10 36 in             
     10 37 1998           
     10 38 .              
     11  1 car            
     11  2 engine         
     11  3 raced          
     11  4 while          
     11  5 slowing        
     11  6 to             
     11  7 park           
     11  8 .              
     11  9 car            
     11 10 lurched        
     11 11 forward        
     11 12 and            
     11 13 crashed        
     11 14 into           
     11 15 a              
     11 16 fence          
     11 17 and            
     11 18 a              
     11 19 building       
     11 20 .              
     11 21 car            
     11 22 had            
     11 23 been           
     11 24 in             
     11 25 shop           
     11 26 approximately  
     11 27 one            
     11 28 week           
     11 29 prior          
     11 30 to             
     11 31 incident       
     11 32 for            
     11 33 high           
     11 34 idle           
     11 35 condition      
     11 36 .              
     12  1 rear           
     12  2 ended          
     12  3 another        
     12  4 vehicle        
     12  5 at             
     12  6 65             
     12  7 to             
     12  8 70mph          
     12  9 and            
     12 10 neither        
     12 11 driver's       
     12 12 side           
     12 13 or             
     12 14 passenger's    
     12 15 side           
     12 16 airbags        
     12 17 deployed       
     12 18 .              
     12 19 dealer         
     12 20 has            
     12 21 vehicle        
     12 22 .              
     13  1 while          
     13  2 vehicle        
     13  3 was            
     13  4 parked         
     13  5 for            
     13  6 an             
     13  7 hour           
     13  8 a              
     13  9 fire           
     13 10 started        
     13 11 on             
     13 12 the            
     13 13 left           
     13 14 side           
     13 15 of             
     13 16 the            
     13 17 engine         
     13 18 compartment    
     13 19 .              
     13 20 owners         
     13 21 son            
     13 22 smelled        
     13 23 smoke          
     13 24 owner          
     13 25 saw            
     13 26 fire           
     13 27 coming         
     13 28 from           
     13 29 around         
     13 30 drivers        
     13 31 side           
     13 32 front          
     13 33 wheel          
     13 34 .              
     13 35 referenced     
     13 36 in             
     13 37 ea02-025       
     14  1 after          
     14  2 vehicle        
     14  3 was            
     14  4 repaired       
     14  5 under          
     14  6 recall         
     14  7 99v029000      
     14  8 ignition       
     14  9 switch         
     14 10 the            
     14 11 airbag         
     14 12 light          
     14 13 stayed         
     14 14 on             
     14 15 .              
     14 16 the            
     14 17 dealer         
     14 18 and            
     14 19 the            
     14 20 manufacturer   
     14 21 has            
     14 22 been           
     14 23 notified       
     14 24 .              
     15  1 electrical     
     15  2 control        
     15  3 module         
     15  4 is             
     15  5 shortening     
     15  6 out            
     15  7 causing        
     15  8 the            
     15  9 vehicle        
     15 10 to             
     15 11 stall          
     15 12 .              
     15 13 engine         
     15 14 will           
     15 15 become         
     15 16 totally        
     15 17 inoperative    
     15 18 .              
     15 19 consumer       
     15 20 had            
     15 21 to             
     15 22 change         
     15 23 alternator/    
     15 24 battery        
     15 25 and            
     15 26 starter        
     15 27 and            
     15 28 module         
     15 29 replaced       
     15 30 4              
     15 31 times          
     15 32 but            
     15 33 defect         
     15 34 still          
     15 35 occurring      
     15 36 cannot         
     15 37 determine      
     15 38 what           
     15 39 is             
     15 40 causing        
     15 41 the            
     15 42 problem        
     15 43 .              
     16  1 at             
     16  2 68000          
     16  3 miles          
     16  4 power          
     16  5 steering       
     16  6 broke          
     16  7 off            
     16  8 the            
     16  9 housing        
     16 10 pump           
     16 11 causing        
     16 12 total          
     16 13 loss           
     16 14 of             
     16 15 power          
     16 16 steering       
     16 17 which          
     16 18 also           
     16 19 caused         
     16 20 the            
     16 21 vehicle        
     16 22 to             
     16 23 shut           
     16 24 down           
     16 25 .              
     17  1 on             
     17  2 two            
     17  3 occasions      
     17  4 dual           
     17  5 airbags        
     17  6 did            
     17  7 not            
     17  8 deploy         
     17  9 .              
     17 10 consumer       
     17 11 rearended      
     17 12 another        
     17 13 vehicle        
     17 14 at             
     17 15 approximately  
     17 16 50             
     17 17 mphand         
     17 18 at             
     17 19 80             
     17 20 mph            
     17 21 hit            
     17 22 a              
     17 23 truck          
     17 24 head-on        
     17 25 upon           
     17 26 impact         
     17 27 air            
     17 28 bags           
     17 29 did            
     17 30 not            
     17 31 deploy         
     17 32 .              
     17 33 driver         
     17 34 sustained      
     17 35 injuries.dealer
     17 36 did            
     17 37 not            
     17 38 determine      
     17 39 why            
     17 40 air            
     17 41 bags           
     17 42 did            
     17 43 not            
     17 44 deploy         
     17 45 .              
     18  1 sunroof        
     18  2 is             
     18  3 leaking        
     18  4 .              
     18  5 yh             
     19  1 motor          
     19  2 and            
     19  3 the            
     19  4 frame          
     19  5 separated      
     19  6 from           
     19  7 vehicle        
     19  8 .              
     19  9 manufacturer   
     19 10 will           
     19 11 be             
     19 12 notified       
     19 13 .              
     20  1 rear           
     20  2 front          
     20  3 wheel          
     20  4 bearing        
     20  5 broke          
     20  6 causing        
     20  7 vehicle        
     20  8 to             
     20  9 pull           
     20 10 to             
     20 11 the            
     20 12 left           
     20 13 when           
     20 14 slowing        
     20 15 down           
     20 16 .              
     20 17 consumer       
     20 18 had            
     20 19 brake's        
     20 20 replaced       
     20 21 about          
     20 22 four           
     20 23 times          
     20 24 and            
     20 25 still          
     20 26 dealer         
     20 27 can't          
     20 28 determine      
     20 29 the            
     20 30 problem        
     20 31 .

Download a zip file of all examples and a SQL script file that creates their input tables from the attachment in the left sidebar.