17.10 - Example: Using the MANIFESTFILE Option - Advanced SQL Engine - Teradata Database

Teradata Vantage™ - Native Object Store Getting Started Guide

Product
Advanced SQL Engine
Teradata Database
Release Number
17.10
Release Date
July 2021
Content Type
Programming Reference
Publication ID
B035-1214-171K
Language
English (United States)

Create a manifest file listing the paths of the objects stored on external storage using the MANIFESTFILE option. MANIFESTFILE creates a manifest file on external object store at the location specified.

  1. Write all rows matching the criteria to external object store and create a manifest file listing the paths of the objects stored on external storage:
    SELECT * FROM WRITE_NOS (
    ON ( SELECT * FROM RiverFlowPerm WHERE DateTime = '2018-06-27 00:00' )
    USING
    LOCATION('YOUR-OBJECT-STORE-URI/20180627/ManifestFile/')
    AUTHORIZATION(MyAuthObj_Write)
    STOREDAS('PARQUET')
    MANIFESTFILE('YOUR-OBJECT-STORE-URI/20180627/ManifestFile/manifest1.json')
    ) AS d;
    

    Replace the LOCATION of YOUR-OBJECT-STORE-URI/20180701/ManifestFile/ with the URI to the external object store where you want to write the data.

    Replace the MANIFESTFILE location of YOUR-OBJECT-STORE-URI/20180701/ManifestFile/manifest1.json with the URI to the manifest file on external object store.

    Your result will be similar to the following:

         NodeId          33
          AmpId           0
       Sequence                    1
     ObjectName /S3/s3.amazonaws.com/iewritenostest/20180627/ManifestFile/object_33_0_1.parquet
     ObjectSize                 1327
    RecordCount                    3
    
         NodeId          33
          AmpId           3
       Sequence                    1
     ObjectName /S3/s3.amazonaws.com/iewritenostest/20180627/ManifestFile/object_33_3_1.parquet
     ObjectSize                 1346
    RecordCount                    3
    [...]
  2. You can view the manifest file on external storage using the command line of your external object store.

    The entries stored in the manifest file are similar to this:

    {"entries":[{"url":"s3://ie-writenos-bucket/20180701/ManifestFile/object_33_0_1.parquet","meta":{"content_length":2803}},
               {"url":"s3://ie-writenos-bucket/20180701/ManifestFile/object_33_6_1.parquet","meta":{"content_length":2733}},
               {"url":"s3://ie-writenos-bucket/20180701/ManifestFile/object_33_1_1.parquet","meta":{"content_length":3009}},
               {"url":"s3://ie-writenos-bucket/20180701/ManifestFile/object_33_7_1.parquet","meta":{"content_length":2591}},
               {"url":"s3://ie-writenos-bucket/20180701/ManifestFile/object_33_2_1.parquet","meta":{"content_length":2725}}]
    }