Term Frequency-Inverse Document Frequency (TF-IDF) is a technique for evaluating the importance of a specific term in a specific document in a document set. Term frequency (tf) is the number of times that the term appears in the document and inverse document frequency (idf) is the number of times that the term appears in the document set. The TF-IDF score for a term is tf * idf. A term with a high TF-IDF score is relevant to the specific document.
You can use the TF-IDF scores as input for documents clustering and classification algorithms, including:
- Cosine-similarity
- Latent Dirichlet allocation
- K-means clustering
- K-nearest neighbors
TD_TFIDF function represents each document as an N-dimensional vector, where N is the number of terms in the document set (therefore, the document vector is sparse). Each entry in the document vector is the TF-IDF score of a term.