READ_NOS Syntax Elements - Analytics Database - Teradata Vantage

connector

Identifies the type of external storage system where the data is located.

Teradata requires the storage location to start with the following for all external storage locations:

• Amazon S3 storage location must begin with /S3 or /s3
• Azure Blob storage location (including Azure Data Lake Storage Gen2 in Blob Interop Mode) must begin with /AZ or /az.
• Google Cloud Storage location must begin with /GS or /gs

storage-account

Used by Azure. The Azure storage account contains your Azure storage data objects.

endpoint

A URL that identifies the system-specific entry point for the external object store system.

bucket (Amazon S3, GCS) or container (Azure Blob storage and Azure Data Lake Storage Gen2)

A container that logically groups stored objects in the external storage system.

key_prefix

Identifies one or more objects in the logical organization of the bucket data. Because it is a key prefix, not an actual directory path, the key prefix may match one or more objects in the external storage. For example, the key prefix '/fabrics/cotton/colors/b/' would match objects: /fabrics/cotton/colors/blue, /fabrics/cotton/colors/brown, and /fabrics/cotton/colors/black. If there were organization levels under those, such as /fabrics/cotton/colors/blue/shirts, the same key prefix would gather those objects too.

Vantage validates only the first file it encounters from the location key prefix.

For example, this LOCATION value might specify all objects on an Amazon cloud storage system for the month of December, 2001:

LOCATION('/S3/YOUR-BUCKET.s3.amazonaws.com/csv/US-Crimes/csv-files/2001/Dec/')

This LOCATION could specify an individual storage object (or file), Day1.csv:

LOCATION('/S3/YOUR-BUCKET.s3.amazonaws.com/csv/US-Crimes/csv-files/2001/Dec/Day1.csv')

This LOCATION specifies an entire container in an Azure external object store (Azure Blob storage or Azure Data Lake Storage Gen2). The container may contain multiple file objects:

LOCATION('/AZ/YOUR-STORAGE-ACCOUNT.blob.core.windows.net/CONTAINER/nos-csv-data')

This is an example of a Google Cloud Storage location:

LOCATION('/gs/storage.googleapis.com/YOUR-BUCKET/CSVDATA/RIVERS/rivers.csv')

AUTHORIZATION = '{"Access_ID":"access_id", "Access_Key":"secret_key" 
,"Session_Token":"session_token" }' 

SCANPCT is enabled by default. To disable it, set DBControl option SCANPCTOption to 1. To re-enable it, set SCANPCTOption to 0 or 2.

• Input table and LOCATION and an empty table.
• Input table with a row for each external file.

• Define an input table with a single column, Payload, with the appropriate data type:
  • JSON

  This column determines the output Payload column return type.

  See Example: Using READ_NOS with NOSREAD_RECORD Return Type.

• For LOCATION, specify the filepath.

• Location VARCHAR(2048) CHARACTER SET UNICODE
• ObjectVersionID VARCHAR(1024) CHARACTER SET UNICODE
• ObjectTimeStamp TIMESTAMP(6)
• OffsetIntoObject BIGINT
• ObjectLength BIGINT
• Payload JSON

This table can be populated using the output of the NOSREAD_KEYS return type.

• Location
• ObjectVersionID
• ObjectTimeStamp
• ObjectLength, size of external file.

See Example: Using READ_NOS with NOSREAD_KEYS Return Type.

{
  "entries": [
     {"url":"s3://nos-core-us-east-1/UNICODE/JSON/mln-key/data-10/data-8_9_02-10.json"},
     {"url":"s3://nos-core-us-east-1/UNICODE/JSON/mln-key/data-10/data-8_9_02-101.json"},
     {"url":"s3://nos-core-us-east-1/UNICODE/JSON/mln-key/data-10/data-10-01/data-8_9_02-102.json"},
     {"url":"s3://nos-core-us-east-1/UNICODE/JSON/mln-key/data-10/data-10-01/data-8_9_02-103.json"}
  ]
}

LOCATION('/S3/YOUR-BUCKET.s3.amazonaws.com/testdeltalake/deltalakewp/_symlink_format_manifest')

LOCATION('/S3/YOUR-BUCKET.s3.amazonaws.com/testdeltalake/deltalakewp/_symlink_format_manifest/zip=95661/manifest')

field_delimiter

The default is "," (comma). You can also specify a custom field delimiter, such as tab "\t".

record_delimiter

New line feed character: "\n". A line feed (\n) is the only acceptable record delimiter.

character_set

"UTF8" or "LATIN"

If you do not specify a ROWFORMAT or payload column, Vantage assumes UTF8 Unicode.