detect_language() examples | Teradata Package for Generative AI - Examples: How to use detect_language() - Teradata VantageCloud Lake

Example setup: Import required packages and set up input data

>>> import os
>>> import teradatagenai
>>> from teradatagenai import TeradataAI, TextAnalyticsAI, load_data
>>> from teradataml import DataFrame
>>> load_data('employee', 'employee_data')
>>> data = DataFrame('employee_data')
>>> df_quotes = data.select(["employee_id", "employee_name", "quotes"])

Complete the example setup requirements.

Example 1: Detect the language of text in the 'quotes' column of a teradataml DataFrame using hugging face model 'xlm-roberta-base-language-detection'

The text for language detection is passed as a column name along with the teradataml DataFrame. A specific language is passed in the 'language' argument.

Create an LLM endpoint.

>>> model_name = 'papluca/xlm-roberta-base-language-detection'
>>> model_args = {'transformer_class': 'AutoModelForSequenceClassification',
                  'task' : 'text-classification'}
>>> ues_args = {'env_name' : 'demo_env'}
>>> llm = TeradataAI(api_type = "hugging_face",
                     model_name = model_name,
                     model_args = model_args,
                     ues_args = ues_args)

Create a TextAnalyticsAI object.

>>> obj = TextAnalyticsAI(llm = llm)

Detect the language of the 'quotes' column in the 'df_quotes' teradataml DataFrame.

>>> obj.detect_language(column = "quotes",
                        data = df_quotes,
                        delimiter = "#")

Output:

Example 2: Extend Example 1 to use default script with 'output_labels' to format the output

>>> obj.detect_language(column = 'quotes',
                        data = df_quotes,
                        output_labels = {'label': str, 'score': float},
                        delimiter = "#")

Output:

Example 3: Extend Example 2 to use user defined script for inference

>>> base_dir = os.path.dirname(teradatagenai.__file__)
>>> language_detection_script = os.path.join(base_dir, 'example-data',
                                             'detect_language.py')

>>> obj.detect_language(column = 'quotes',
                        data = df_quotes,
                        script = language_detection_script,
                        delimiter = "#")

Output:

obj_nim.detect_language(column="quotes",data=data,volatile=True)

Output: