ChunkText Input - Teradata Vantage

Teradata Vantage™ - Bring Your Own Model User Guide

Deployment

VantageCloud

VantageCore

Edition

VMware

Enterprise

IntelliFlex

Lake

Product

Teradata Vantage

Release Number

20.00

Published

May 2026

ft:locale

en-US

ft:lastEdition

2026-06-18

dita:mapPath

fee1607120608274.ditamap

dita:ditavalPath

ayr1485454803741.ditaval

dita:id

fee1607120608274

Input Table	Description
InputTable	Table that contains a column named txt (VARCHAR or CLOB) with input text to be split into chunks. Table can include additional columns. Supports Unicode input data.
TokenizerTable	Table that contains the following columns: tokenizer_id tokenizer (BLOB): Contains the contents of the tokenizer.json file. When TokenizerTable contains more than one tokenizer, you must use a SELECT query with a WHERE clause to select the tokenizer, otherwise an error occurs. TokenizerTable must be specified as DIMENSION, otherwise an error occurs.

Input Table

Description

InputTable

Table that contains a column named txt (VARCHAR or CLOB) with input text to be split into chunks.

Table can include additional columns.

Supports Unicode input data.

TokenizerTable

Table that contains the following columns:

tokenizer_id
tokenizer (BLOB): Contains the contents of the tokenizer.json file.

When TokenizerTable contains more than one tokenizer, you must use a SELECT query with a WHERE clause to select the tokenizer, otherwise an error occurs.

TokenizerTable must be specified as DIMENSION, otherwise an error occurs.