TFIDF Example 2: Tokenized Test Set - Teradata Vantage

This example uses the IDF values from tfidf_output1, output by TFIDF Example 1: Tokenized Training Document Set to predict the TFIDF scores of a test document set.

NGrams Input: tfidf_test

NGrams SQL Call

This call creates a table of tokenized input, tfidf_token1, from tfidf_test.

CREATE MULTISET TABLE tfidf_token1 AS (
  SELECT * FROM NGrams (
    ON tfidf_test
    USING
    TextColumn ('content')
    Delimiter (' ')
    Grams ('1')
    Overlapping ('false')
    ToLowerCase ('true')
    Punctuation ('\[.,?\!\]')
    Reset ('\[.,?\!\]')
    TotalGramCount ('false')
    Accumulate ('docid')
  ) AS dt
) WITH DATA;

SQL Call to Create TFIDF Input Table tfidf_input1

CREATE MULTISET TABLE tfidf_input1 AS (
  SELECT docid, ngram AS term, frequency AS "count" FROM tfidf_token1 AS dt
) WITH DATA;

SQL Call to Create TFIDF Input Table tf1

CREATE MULTISET TABLE tf1 AS (
  SELECT * FROM tf (
    ON tfidf_input1 PARTITION BY docid
    USING
    Formula ('normal')
  ) AS dt1
) WITH DATA;

TFIDF SQL Call

CREATE MULTISET TABLE tfidf_output1 AS (
  SELECT * FROM TFIDF (
    ON tf1 AS tf PARTITION BY TERM
	ON (SELECT CAST(COUNT(DISTINCT(docid)) AS INTEGER) AS "count"
      FROM tfidf_input1) AS doccount DIMENSION
  ) AS dt
) WITH DATA;

TFIDF Output

This query returns the following table:

SELECT * FROM tfidf_output1;