Introduction to Native Object Store Functionality| VantageCloud Lake - NOS Functionality - Teradata Vantage

Teradata® VantageCloud Lake
Deployment
VantageCloud
Edition
Lake
Product
Teradata Vantage
Published
January 2023
ft:locale
en-US
ft:lastEdition
2024-12-11
dita:mapPath
phg1621910019905.ditamap
dita:ditavalPath
pny1626732985837.ditaval
dita:id
phg1621910019905
NOS enables you to do the following:
  • Analyze data stored on external object storage
  • Read data in CSV, JSON, or Parquet format from external object storage
  • Join or aggregate external data to relational data stored in Analytics Database
  • Query cold data offloaded to external object storage
  • Load data from an external object storage into the database using one SQL request
  • Write Analytics Database data to external object storage. The data to be written can come from a table, derived results, another external object storage, QueryGrid federated query, and so on.
Reading data from external object storage:
Foreign Tables
Users with CREATE TABLE privilege can create a foreign table inside the database, point this virtual table to an external object storage location, and use SQL to translate the external data into a form useful for business.
Using a foreign table in Analytics Database gives you the ability to:
  • Load external data to the database
  • Join external data to data stored in the database
  • Filter the data
  • Use views to simplify how the data appears to your users
  • Use Delta Lake manifest files
Data read through a foreign table is not automatically stored on disk and the data can only be seen by that query. Data can be loaded into the database by accessing a foreign table using these commands: CREATE TABLE AS ... WITH DATA, CREATE TABLE AS … FROM READ_NOS, and INSERT ... SELECT.
READ_NOS
READ_NOS allows you to do the following:
  • Perform an ad hoc query on all data formats with the data in-place on external object storage
  • List all the objects and path structure of external object storage
  • List the object storage
  • Discover the schema of the data
  • Read CSV, JSON, and Parquet data
  • Bypass creating a foreign table in the Analytics Database
  • Load data into the database with INSERT … SELECT where the select references READ_NOS
  • Use a foreign table to query data stored by READ_NOS
  • Use Delta Lake manifest files
Writing data to external object storage:
WRITE_NOS
WRITE_NOS allows you to write data from database tables to external object storage in Parquet format. Data stored by WRITE_NOS can be queried using a foreign table and READ_NOS.
WRITE_NOS allows you to do the following:
  • Extract selected or all columns from an Analytics Database table or from derived results and write to external object storage in Parquet data format.
  • Write to Teradata-supported external object storage, such as Amazon S3.
  • Load data into the database with INSERT ... SELECT where the select references WRITE_NOS
  • Use a foreign table to query data stored by WRITE_NOS

Supported External Object Storage Platforms

At the time of printing of this guide, the following external object storage platforms are supported:
  • Amazon S3
  • Azure Data Lake Storage Gen2
  • Cloudera Ozone
  • Cloudian HyperStore
  • Cohesity SmartFiles
  • Dell ECS Object Store
  • Google Cloud Storage
  • Hitachi Content Platform
  • IBM Cloud Object Store (IBM COS)
  • Microsoft Azure Blob storage
  • MinIO
  • NetApp StorageGRID
  • RedHat Ceph
  • Scality Ring
  • VAST DataStore
  • Vcinity VAccess

Supported Compression Formats

External data may arrive from an object in compressed format. Such data is decompressed inside the Analytics Database, but only after decryption has been completed on the external object storage before being transmitted. GZIP is the only compression format supported for both JSON and CSV. Brotli, Snappy, and Zstd are supported for Parquet. The database recognizes the ".gz" suffix on the incoming files and performs the decompression automatically. Note, compression may bring some trade-offs, such as CPU overhead versus reduced needed Bandwidth amongst others.

Encryption

To encrypt files written to external object storage, configure the destination bucket to encrypt all objects using server-side encryption. Server-side encryption at the bucket level is supported by WRITE_NOS, READ_NOS, and foreign tables.

Note, all data is transmitted between the Vantage platform and the external object storage using TLS encryption, independent of whether the data is encrypted at rest in external object storage.