These examples demonstrate the process of finding the similarity score between the employee data and articles gathered by various employees stored in a TeradataML DataFrame.
Example setup: Import the modules and create a teradataml DataFrame
>>> import os
>>> import teradatagenai
>>> from teradatagenai import TeradataAI, TextAnalyticsAI, load_data
>>> from teradataml import DataFrame
>>> load_data('employee', 'employee_data')
>>> data = DataFrame('employee_data')
Example 1: Get the similarity score for 'employee_data' and 'articles' columns using hugging face model: 'sentence-transformers/all-MiniLM-L6-v2'
Create an LLM endpoint.
>>> model_name = 'sentence-transformers/all-MiniLM-L6-v2'
>>> model_args = {'transformer_class': 'AutoModel',
'task': 'token-classification'}
>>> llm = TeradataAI(api_type = "hugging_face",
model_name = model_name,
model_args = model_args)
Create a TextAnalyticsAI object.
>>> obj = TextAnalyticsAI(llm=llm)
>>> obj.sentence_similarity(column1 = "employee_data",
column2 = "articles",
data = data,
libs = 'sentence_transformers',
delimiter = "#")
Output:
| sentence1 | sentence2 | similarity_score |
|---|---|---|
| Parker Doe originally from Brazil has successfully cleared all their loans by 2020 04 25 Reach them | Climate change poses significant challenges globally affecting weather patterns ecosystems and human | 0.6499 |
| Michael Brown an Australian has a loan due on 2023 07 20 Contact them at 555 555 5558 Their SSN is 6 | The recent 2020 United States election have shown a shift in the political landscape With more youn | 0.7055 |
| Emily Johnson from the UK cleared all their loans by 2021 01 15 They can be contacted at 555 555 555 | Renewable energy sources such as solar wind and hydroelectric power play a crucial role in reducing | 0.6245 |
| Linda Taylor a German cleared all their loans by 2020 12 31 They can be reached at 555 555 5559 Thei | The 2019 Amazon rainforest wildfires were a severe environmental crisis The fires burned thousands o | 0.6725 |
| Alex Smith a Canadian has an outstanding loan due on 2022 05 30 Their contact number is 555 555 5556 | The 2020 Tokyo Olympics was postponed to 2021 due to the COVID-19 pandemic. This was the first time i | 0.7650 |
Example 2: Extend Example 1 to use user defined script for inferencing
>>> base_dir = os.path.dirname(teradatagenai.__file__)
>>> sentence_similarity_script = os.path.join(base_dir, 'example-data', 'sentence_similarity.py')
>>> returns = {"sentence1": VARCHAR(10000),
"sentence2": VARCHAR(10000),
"similarity_score": VARCHAR(10000)}
>>> obj.sentence_similarity(column1 = "employee_data",
column2 = "articles",
data = data,
script = sentence_similarity_script,
returns = returns,
libs = 'sentence_transformers',
delimiter = "#")
Output:
| sentence1 | sentence2 | similarity_score |
|---|---|---|
| Parker Doe originally from Brazil has successfully cleared all their loans by 2020 04 25 Reach them | Climate change poses significant challenges globally affecting weather patterns ecosystems and human | 0.0116 |
| Michael Brown an Australian has a loan due on 2023 07 20 Contact them at 555 555 5558 Their SSN is 6 | The recent 2020 United States election have shown a shift in the political landscape With more youn | 0.0427 |
| Alex Smith a Canadian has an outstanding loan due on 2022 05 30 Their contact number is 555 555 5556 | The 2020 Tokyo Olympics was postponed to 2021 due to the COVID-19 pandemic. This was the first time i | 0.0641 |
| Emily Johnson from the UK cleared all their loans by 2021 01 15 They can be contacted at 555 555 555 | Renewable energy sources such as solar wind and hydroelectric power play a crucial role in reducing | 0.0427 |
| Linda Taylor a German cleared all their loans by 2020 12 31 They can be reached at 555 555 5559 Thei | The 2019 Amazon rainforest wildfires were a severe environmental crisis The fires burned thousands o | 0.0588 |
Example 3: Get the similarity score for 'employee_data' and 'articles' columns using hugging face model: 'distilbert-base-uncased'
>>> model_name = 'distilbert/distilbert-base-uncased'
>>> model_args = {'transformer_class': 'DistilBertModel'}
>>> llm = TeradataAI(api_type = "hugging_face",
model_name = model_name,
model_args = model_args)
>>> obj = TextAnalyticsAI(llm=llm)
>>> obj.sentence_similarity(column1 = "employee_data",
column2 = "articles",
data = data,
libs = 'sentence_transformers',
delimiter = "#")
Output:
| sentence1 | sentence2 | similarity_score |
|---|---|---|
| Alex Smith a Canadian has an outstanding loan due on 2022 05 30 Their contact number is 555 555 5556 | The 2020 Tokyo Olympics was postponed to 2021 due to the COVID-19 pandemic. This was the first time i | 0.0641 |
| Linda Taylor a German cleared all their loans by 2020 12 31 They can be reached at 555 555 5559 Thei | The 2019 Amazon rainforest wildfires were a severe environmental crisis The fires burned thousands o | 0.0588 |
| Emily Johnson from the UK cleared all their loans by 2021 01 15 They can be contacted at 555 555 555 | Renewable energy sources such as solar wind and hydroelectric power play a crucial role in reducing | 0.0427 |
| Parker Doe originally from Brazil has successfully cleared all their loans by 2020 04 25 Reach them | Climate change poses significant challenges globally affecting weather patterns ecosystems and human | 0.0116 |
| Michael Brown an Australian has a loan due on 2023 07 20 Contact them at 555 555 5558 Their SSN is 6 | The recent 2020 United States election have shown a shift in the political landscape With more youn | 0.0427 |