Setting up Vantage to Delta Lake Integration - Analytics Database - Teradata Vantage

Teradata Vantage™ - Native Object Store Getting Started Guide - 17.20

Deployment
VantageCloud
VantageCore
Edition
Enterprise
IntelliFlex
VMware
Product
Analytics Database
Teradata Vantage
Release Number
17.20
Published
June 2022
Language
English (United States)
Last Update
2024-04-05
dita:mapPath
tsq1628112323282.ditamap
dita:ditavalPath
qkf1628213546010.ditaval
dita:id
jjn1567647976698
Product Category
Teradata Vantage

To set up a Vantage to Delta Lake integration using manifest files and query Delta tables:

    Generate Manifests of a Delta Table

  1. See https://docs.delta.io/latest/integrations.html for instructions on how to generate and configure manifests of a Delta table.
  2. Configure Vantage to Read the Generated Manifests

  3. Define a new foreign table in Vantage using the format SymlinkTextInputFormat and the manifest location path-to-delta-table/_symlink_format_manifest/:
    CREATE FOREIGN TABLE mytable
        , EXTERNAL SECURITY myauthobj
        USING (
        LOCATION('/s3/mybucket.s3.amazonaws.com/path-to-delta=table/_symlink_format_manifest/')
        MANIFEST('TRUE')
        TABLE_FORMAT('DELTALAKE') );
    The presence of the _symlink_format_manifest keystring confirms that it is a Delta Lake table and is validated only if the MANIFEST parameter is set to TRUE and the TABLE_FORMAT parameter is set as DELTALAKE.
    SymlinkTextInputFormat instructs Vantage to read the data for mytable by reading the manifest file instead of using a directory listing to find data files. Replace mytable with the name of the external table and path-to-delta-table with the absolute path to the Delta table.
    Even though Delta Lake supports schema evolution and queries on a Delta table automatically use the latest schema regardless of the schema defined in the table, Vantage uses the schema defined in its foreign table definition and does not query with the updated schema until the table definition is updated to the new schema.