1.0 - 8.00 - TFIDF Syntax - Teradata Vantage

Teradata® Vantage Machine Learning Engine Analytic Function Reference

Product
Teradata Vantage
Release Number
1.0
8.00
Release Date
May 2019
Content Type
Programming Reference
Publication ID
B700-4003-098K
Language
English (United States)

TFIDF version 2.3, TF version 1.2

SELECT * FROM TFIDF (
  ON TF (
    ON { table | view | (query) } PARTITION BY docid     
    [ USING Formula ({ 'normal' | 'bool' | 'log' | 'augment' }) ]
  ) AS tf PARTITION BY term 
  [ ON (SELECT COUNT (DISTINCT docid) FROM doccount_table) AS doccount DIMENSION ]
  [ ON (SELECT term, COUNT (DISTINCT docid) FROM docperterm_table GROUP BY term)
      AS docperterm PARTITION BY term
  ]
  [ ON (SELECT DISTINCT (term) AS term, idf FROM tf_idf_output_table)
      AS idf PARTITION BY term
  ]
) AS alias;

Large Document Sets

For large documents sets, the docperterm_table is required.

For training, this is the syntax for large document sets:

SELECT * FROM TFIDF (
  ON TF (
    ON { table | view | (query) } PARTITION BY docid      
    [ USING Formula ({ 'normal' | 'bool' | 'log' | 'augment' }) ]
  ) AS tf PARTITION BY term 
  ON (SELECT COUNT (DISTINCT docid) FROM doccount_table) AS doccount DIMENSION
  ON (SELECT term, COUNT (DISTINCT docid) FROM docperterm_table GROUP BY term)
    AS docperterm PARTITION BY term 
) AS alias ORDER BY docid;

For prediction, this is the syntax for large document sets:

SELECT * FROM TFIDF (
  ON TF (
    ON { table | view | (query) } PARTITION BY docid      
    [ USING Formula ({ 'normal' | 'bool' | 'log' | 'augment' }) ]
  ) AS tf PARTITION BY term 
  [ ON (SELECT term, COUNT (DISTINCT docid) FROM docperterm_table GROUP BY term)
      AS docperterm PARTITION BY term
  ]
  [ ON (SELECT DISTINCT (term) AS term, idf FROM tf_idf_output_table)
      AS idf PARTITION BY term
  ]
) AS alias ORDER BY docid;

Small Document Sets

This syntax is acceptable for small document sets:

SELECT * FROM TFIDF (
  ON TF (
    ON { table | view | (query) } PARTITION BY docid 
  ) AS tf PARTITION BY term 
  ON (SELECT COUNT (DISTINCT docid) FROM input_table) AS doccount DIMENSION
) AS alias ORDER BY docid;