mask_pii() examples | Teradata Package for Generative AI - Examples: How to use mask_pii - Teradata VantageCloud Lake

Example setup: Import the modules and create a teradataml DataFrame

>>> import os
>>> import teradatagenai
>>> from teradatagenai import TeradataAI, TextAnalyticsAI, load_data
>>> from teradataml import DataFrame
>>> load_data('employee', 'employee_data')
>>> data = DataFrame('employee_data')
>>> df_employeeData = data.select(["employee_id", "employee_name", "employee_data"])

Complete the example setup requirements.

Example 1: Recognize PII entities in the 'employee_data' column of a teradataml DataFrame using hugging face model 'lakshyakh93/deberta_finetuned_pii'

The text containing potential PII like names, addresses, credit card numbers, etc., is passed as a column name along with the teradataml DataFrame. Setting the 'internal_mask' as True indicates masking to be done by the inbuilt function.

Create an LLM endpoint.

>>> model_name = 'lakshyakh93/deberta_finetuned_pii'
>>> model_args = {'transformer_class': 'AutoModelForTokenClassification',
                  'task' : 'token-classification'}
>>> llm = TeradataAI(api_type = "hugging_face",
                     model_name = model_name,
                     model_args = model_args)

Create a TextAnalyticsAI object.

>>> obj = TextAnalyticsAI(llm = llm)
>>> obj.mask_pii(column="employee_data",
                 data=df_employeeData,
                 delimiter="#",
                 internal_mask=True)

Output:

Example 2: Extend Example 1 to use user defined script for masking

>>> base_dir = os.path.dirname(teradatagenai.__file__)
>>> mask_pii_script = os.path.join(base_dir, 'example-data',
                                   'mask_pii.py')
>>> obj.mask_pii(column = "employee_data",
                 data = df_employeeData,
                 script = mask_pii_script,
                 delimiter = "#")

Output:

obj_nim.mask_pii(column="employee_data",data=data,accumulate='employee_data',volatile=True)

Output: