LDA Example - Teradata Vantage

Input

The input (training) table is log of vehicle complaints. The category column indicates whether the car has been in a crash.

The stop words table, stopwords.text, contains:

a
an
in
is
to
into
was
the
and
this
with
they
but
will

To create a tokenized, filtered input file for the LDA function, apply the function TextParser to the training table:

CREATE MULTISET TABLE complaints_traintoken AS (
  SELECT * FROM TextParser (
    ON complaints
    USING
    TextColumn ('text_data')
    ToLowerCase ('true')
    Stemming ('false')
    ListPositions ('true')
    StopWords ('stopwords.txt')
    RemoveStopWords ('true')
    Accumulate ('doc_id', 'category')
  ) AS dt
) WITH DATA;

This query returns the following table:

SELECT * FROM complaints_traintoken ORDER BY doc_id;

SQL Call

SELECT * FROM LDA (
  ON complaints_traintoken AS InputTable
  OUT TABLE ModelTable (ldamodel)
  OUT TABLE OutputTable (ldaout1)
  USING
  TopicNum (5)
  DocIDColumn ('doc_id')
  WordColumn ('token')
  CountColumn ('frequency')
  MaxIterNum (30)
  ConvergenceDelta (1e-3)
  Seed (2)
) AS dt;

Output

Outputtable is created successfully.
Training converged after 7 iterate steps with delta 1.4006041962568355E-4
There are 20 documents with 520 words in the training set, the perplexity is 92.070615

This query returns the following table:

SELECT * FROM ldaout1 ORDER BY docid, topicid;