17.10 - Setting up Vantage to Delta Lake Integration - Advanced SQL Engine - Teradata Database

Teradata Vantage™ - Native Object Store Getting Started Guide

Product
Advanced SQL Engine
Teradata Database
Release Number
17.10
Release Date
July 2021
Content Type
Programming Reference
Publication ID
B035-1214-171K
Language
English (United States)

To set up a Vantage to Delta Lake integration using manifest files and query Delta tables:

    Generate Manifests of a Delta Table

  1. See https://docs.delta.io/latest/integrations.html for instructions on how to generate and configure manifests of a Delta table.
  2. Configure Vantage to Read the Generated Manifests

  3. Define a new foreign table in Vantage using the format SymlinkTextInputFormat and the manifest location path-to-delta-table/_symlink_format_manifest/:
    CREATE FOREIGN TABLE mytable
        , EXTERNAL SECURITY myauthobj
        USING (
        LOCATION('/s3/mybucket.s3.amazonaws.com/path-to-delta=table/_symlink_format_manifest/')
        MANIFEST('TRUE')
        TABLE_FORMAT('DELTALAKE') );
    The presence of the _symlink_format_manifest keystring confirms that it is a Delta Lake table and is validated only if the MANIFEST parameter is set to TRUE and the TABLE_FORMAT parameter is set as DELTALAKE.
    SymlinkTextInputFormat instructs Vantage to read the data for mytable by reading the manifest file instead of using a directory listing to find data files. Replace mytable with the name of the external table and path-to-delta-table with the absolute path to the Delta table.
    Even though Delta Lake supports schema evolution and queries on a Delta table automatically use the latest schema regardless of the schema defined in the table, Vantage uses the schema defined in its foreign table definition and does not query with the updated schema until the table definition is updated to the new schema.