This is the Aster Scoring SDK version of the TextTokenizer function, which is discussed in TextTokenizer.