Use case: You want to break down a human-readable text into machine understandable language.
Tokenization is a technique used to represent text data into a numeric format that can then be used to train a neural network. It involves breaking down text into smaller units known as tokens (words, characters, n-grams) and then associating them with a unique integer index. These characteristics make tokenization a popular technique in the field of Natural Language Processing.
There are different kinds of tokenizers that can be demonstrated using the tokenizers package. Following is a simple usage of tokenizing into character-based shingles.
tokenize_character_shingles( x, n = 3L, n_min = n, lowercase = TRUE, strip_non_alphanum = TRUE, simplify = FALSE )
For this task, assume:
- The tokenizer R script "tokenizers.R" is stored on a client’s machine.
- The requirement file “reqs_tokenizers_R.txt” is stored on client’s machine.
Prerequisite steps:
- Connect from a client to a target VantageCloud Lake system where the task will be performed.
- Import necessary modules on the client for the current use case.
from teradatasqlalchemy.types import VARCHAR
- Specify as a variable the path where the script and model files are kept on the client.
path_to_files = '/Users/JaneDoe/OpeanAFexamples/scripts/'
- Specify as a variable the path where the requirement file is stored on the client.
path_to_reqsfiles = '/Users/JaneDoe/OpeanAFexamples/reqsfiles/'
At present. you can only use a requirement file to install R libraries.