ChunkText is a table operator that splits large text into smaller, token-limited segments called chunks. It uses a HuggingFace tokenizer to count tokens and ensures that no chunk exceeds a specified token limit. ChunkText is available as part of the BYOM package from version 20.00.31.00 and later.
Unlike character-count or word-count splitting, ChunkText uses the actual tokenizer to measure each chunk. This approach produces accurate, model-compatible splits.
When to use ChunkText
Language models have fixed input-size limits measured in tokens. When your source text exceeds that limit, you must split it into chunks that fit within the model's context window. Accurate splitting requires knowing how the model's tokenizer maps text to tokens.
With ChunkText, you can process text directly in the database without extracting data to an external tool.
- Reduces data movement: Text stays in the database and ChunkText processes it where it resides.
- Ensures token-accurate splitting: ChunkText measures chunks with the actual tokenizer, not approximate word or character estimates.
- Prepares data for scoring functions: You can pass chunked output directly to BYOM functions such as ONNXEmbeddings for embedding generation.
How ChunkText works
You load a HuggingFace tokenizer.json file as a BLOB into a tokenizer table. You then run a SELECT query that passes your text data and the tokenizer to ChunkText. The operator splits each row's text into one or more chunks within your specified token limit and returns the results as rows.