WordEmbeddings
Description
The td_word_embeddings_sqle()
function produces vectors for each piece of text and
finds the similarity between the texts.
Word embedding is the representation of a word in multi-dimensional
space such that words with similar meanings have similar embedding.
Each word is mapped to a vector of real numbers that represent the word.
The function contains training and prediction using models. The models contain each possible token and its corresponding vectors. The closer the distance between the vectors the more the similarity. Function operations are token-embedding, doc-embedding, token2token-similarity, and doc2doc-similarity.
Usage
td_word_embeddings_sqle (
data = NULL,
model = NULL,
id.column = NULL,
model.text.column = NULL,
model.vector.columns = NULL,
primary.column = NULL,
secondary.column = NULL,
operation = "token-embedding",
accumulate = NULL,
convert.to.lowercase = TRUE,
remove.stopwords = FALSE,
stem.tokens = FALSE,
...
)
Arguments
data |
Required Argument. |
model |
Required Argument. |
id.column |
Required Argument. |
model.text.column |
Required Argument. |
model.vector.columns |
Required Argument. |
primary.column |
Required Argument. |
secondary.column |
Optional Argument.
Types: character |
operation |
Optional Argument. Types: character |
accumulate |
Optional Argument.
Types: character OR vector of Strings (character) |
convert.to.lowercase |
Optional Argument. |
remove.stopwords |
Optional Argument.
Default Value: FALSE |
stem.tokens |
Optional Argument. |
... |
Specifies the generic keyword arguments SQLE functions accept. Below volatile: Function allows the user to partition, hash, order or local order the input data. These generic arguments are available for each argument that accepts tbl_teradata as input and can be accessed as:
Note: |
Value
Function returns an object of class "td_word_embeddings_sqle"
which is a named list containing object of class "tbl_teradata".
Named list member(s) can be referenced directly with the "$" operator
using the name(s):result
Examples
# Get the current context/connection.
con <- td_get_context()$connection
# Load the example data.
loadExampleData("tdplyr_example", "word_embed_model",
"word_embed_input_table1","word_embed_input_table2")
# Create tbl_teradata object.
model_input <- tbl(con, "word_embed_model")
data_input1 <- tbl(con, "word_embed_input_table1")
data_input2 <- tbl(con, "word_embed_input_table2")
# Check the list of available analytic functions.
display_analytic_functions()
# Example 1 : Generate vectors for each words present in column 'doc1'
# using the word embedding model and 'token-embedding' operation.
# Each word is assigned with vectors and closer the distance
# between two vectors greater the similarity between two words.
WordEmbeddings_out1 <- td_word_embeddings_sqle(
data = data_input1,
model = model_input,
id.column = "doc_id",
model.text.column = "token",
model.vector.columns = c("v1", "v2", "v3", "v4"),
primary.column = "doc1",
operation = "token-embedding")
# Print the result.
print(WordEmbeddings_out1$result)
# Example 2 : Generate vectors for each row present in column 'doc1'
# using the word embedding model and 'doc-embedding' operation.
# Each row with para or sentence is assigned with vectors and
# closer the distance between two vectors greater the similarity
# between two para or sentence.
WordEmbeddings_out2 <- td_word_embeddings_sqle(
data = data_input1,
model = model_input,
id.column = "doc_id",
model.text.column = "token",
model.vector.columns = c("v1", "v2", "v3", "v4"),
primary.column = "doc1",
operation = "doc-embedding",
accumulate = "doc1")
# Print the result.
print(WordEmbeddings_out2$result)
# Example 3 : Find the similarity between two columns 'token1' and 'token2' in 'data_input2',
# using the word embedding model and 'token2token-similarity' operation.
WordEmbeddings_out3 <- td_word_embeddings_sqle(
data = data_input2,
model = model_input,
id.column = "token_id",
model.text.column = "token",
model.vector.columns = c("v1", "v2", "v3", "v4"),
primary.column = "token1",
secondary.column = "token2",
operation = "token2token-similarity",
accumulate = c("token1", "token2"))
# Print the result.
print(WordEmbeddings_out3$result)
# Example 4 : Find the similarity between two columns 'doc1' and 'doc2' in 'data_input1',
# using the word embedding model and 'doc2doc-similarity' operation.
WordEmbeddings_out4 <- td_word_embeddings_sqle(
data = data_input1,
model = model_input,
id.column = "doc_id",
model.text.column = "token",
model.vector.columns = c("v1", "v2", "v3", "v4"),
primary.column = "doc1",
secondary.column = "doc2",
operation = "doc2doc-similarity",
accumulate = c("doc1", "doc2"))
# Print the result.
print(WordEmbeddings_out4$result)