ChunkText Example: Split Text into Chunks - Teradata Vantage

This example shows how to use ChunkText to split text into chunks and store the results in a table.

The query creates a table that contains chunked text, where each row represents a chunk.

create table complaints_chunked as (
    select id, chunk_number, txt, chunk_token_length from
    td_mldb.ChunkText(
        on (select id, txt from complaints_clean_10)
        on (select tokenizer from chunk_tokenizers_tst where tokenizer_id = 'bge-small-en-v1.5') DIMENSION
        using
            MaxTokens(512)
            MaxOverlapWords(10)
            SplittingStrategy('SENTENCES')
            OutputLength(6000)
    ) a
) with data primary index(id, chunk_number);

 *** Table has been created.
 *** Total elapsed time was 1 second.

The query returns chunked text with token length for each chunk.

select * from complaints_chunked order by id, chunk_number;

 *** Query completed. 47 rows found. 4 columns returned. 
 *** Total elapsed time was 1 second.

         id chunk_number chunk_token_length txt
----------- ------------ ------------------ -------------------------------
     701416            0                403 Unknown created a fraudulent RV
     701416            1                467 he was born ( XXXX XXXX XXXX )
     701416            2                400 XXXX, there prior XXXX XXXX hom
     701416            3                477 has XXXX XXXX of XXXX and XXXX
     701416            4                427 a XXXX XXXX FBI background chec
     701617            0                511 I have been contacted by 3 diff
     701617            1                512  1006.34 ( c ) ( 3 ) ( i ) * CF
     701617            2                469 privacy and the consumer themse
     701617            3                255 showing that they are breaking
    1300688            0                411 XX/XX/XXXX Received email payme
    1300688            1                496 by the XXXX XXXX XXXX XXXX XXXX
    1300688            2                217 zone, source IP address, destin
    1300688            3                467 or services hosted on XXXX XXXX
    1300688            4                496 have been a victim of potential
    1300688            5                153 now viewed as more of a credit
    1400062            0                479 STATEMENT OF FACTS Background :
    1400062            1                505 not seem to fully understand ho
    1400062            2                487 whole series of problems as is
    1400062            3                501 XXXX that I must contact Wells
    1400062            4                 40 to the point that they can even
    1501072            0                417 I was selling {200.00} snow tir
    1501072            1                450 ( no subject ) Inbox zelle XXXX
    1501072            2                495 XXXX jonred the call. The name
    1501072            3                418 credited with the total sum of
    1501072            4                136 XXXX XXXX ET, except XXXX, XXXX
    1701663            0                502 I am a XXXX XXXX XXXX ( a XXXX
    1701663            1                509 to first complete their investi
    1701663            2                512 investigator from XX/XX/XXXX. S
    1701663            3                464 told me he explained I dont hav
    1800567            0                453 XX/XX/XXXX XXXX XXXX XXXX XXXX
    1800567            1                495 at least two written requests t
    1800567            2                420 the 90 days stated in XXXX XXXX
    1800567            3                481 How can that be? How do you do
    1800567            4                306 of {$70000.00} which was quoted
    1900140            0                505 XX/XX/XXXX Property Address : X
    1900140            1                505 subsidiary MERS, Countrywide as
    1900140            2                504 the member as a certifying XXXX
    1900140            3                484 is attempting to stand XXXX let
    1901066            0                512 To whom it may concern : I'll g
    1901066            1                459 a {$50.00} late fee charge on t
    1901066            2                475 the month, giving the payment 9
    1901066            3                497 and concrete proof that Comenit
    2200411            0                504 XXXX, XX/XX/XXXX Case : Hacking
    2200411            1                487 office of your Money Market Sav
    2200411            2                497 XXXX, No. XXXX For FixedTerm CD
    2200411            3                497 to whom we supposedly make paym
    2200411            4                143 attention of the department in