| Function | Description |
|---|---|
| Latent Dirichlet Allocation (LDA) Functions | Build a topic model based on the supplied training data and parameters, estimate the topic distribution for each document based on the generated model, and display information from the model. The LDA functions are LDATrainer, LDAInference, and LDATopicPrinter. |
| Levenshtein Distance (LDist) | Computes the Levenshtein distance between two text values, that is, the number of edits needed to transform one string into the other, where edits include insertions, deletions, or substitutions of individual characters. |
| Naive Bayes Text Classifier | Uses the Naive Bayes algorithm to classify data objects. The Naive Bayes Text Classifier is composed of the functions NaiveBayesTextClassifierTrainer and NaiveBayesTextClassifierPredict. |
| Named Entity Recognition (NER) Functions | Use named entity recognition
(NER) to extract features (such as person,
location, and organization) when training data
models, using either the Conditional Random Fields
(CRF) or Max Entropy model. The CRF implementation model functions are NERTrainer, NER, and NEREvaluator. The max entropy model functions are FindNamedEntity, TrainNamedEntityFinder, and Evaluate Named Entity Finder. |
| nGram | Tokenizes (splits) an input stream and emits n multigrams, based on specified delimiter and reset parameters. Useful for sentiment analysis, topic identification, and document classification. |
| POSTagger | Tags the parts-of-speech of input text. |
| Sentenizer | Extracts the sentences in the input paragraphs. |
| Sentiment Extraction Functions | Deduce user opinion (positive, negative, or neutral) from text. The sentiment extraction functions are TrainSentimentExtractor, ExtractSentiment, and EvaluateSentimentExtractor. |
| Text Classifier | Chooses the correct class label for given text. Text Classifier is composed of the functions TextClassifierTrainer, TextClassifier, and TextClassifierEvaluator. |
| TextChunker | Divides text into phrases and assigns each phrase a tag identifying its type. |
| TextMorph | Provides lemmatization, a basic tool in text analysis. Outputs a standard form of the input words. |
| Text_Parser | Tokenizes a stream of words, optionally stems them, and outputs the individual words and their counts. |
| TextTagging | Tags input tuples according to user-defined rules that use logical and text processing operators. |
| TextTokenizer | Extracts tokens (for example, words, punctuation marks, and numbers) from text. |
| TF_IDF | Evaluates the importance of a word within a specific document, weighted by the number of times the word appears in the entire document set. |