Use the aa.text.tokenizer() function to extract tokens (or text segments) from text. Examples of tokens are words, punctuation marks, and numbers. This function supports the following languages: English, Chinese, and Japanese.
Use the aa.text.tokenizer() function to extract tokens (or text segments) from text. Examples of tokens are words, punctuation marks, and numbers. This function supports the following languages: English, Chinese, and Japanese.