The DATALAKE object contains all the required information to connect to an OTF datalake, including authorization credentials for accessing the REST catalog and object storage, as well as the necessary connection details.
OTF supports REST catalogs like Gravitino and Polaris. Gravitino works with AWS and S3-compatible on-prem storage, while Polaris supports AWS only.
The following example shows how to create a DATALAKE that connects to either Polaris or Gravitino. Separate authorization objects are required for the catalog and storage.
The USING clause accepts the following name-value pairs:
| Name | Required | Value |
|---|---|---|
| catalog_type | Y | It indicates the type of the catalog catalog_type(‘rest’) |
| catalog_location | Y | REST Catalog Endpoint URL Example: catalog_location ('http://cnc-tdvm-dmp-8878.8181/api.catalog ') |
| catalog_name | Mandatory for Polaris | REST Catalog Name catalog_name('tdotf_aws_polaris_catalog') |
| idp_type | Optional if OAuth is custom which is of grant_type=client_credentials By default, it is custom. Currently, this is the only supported value. |
Identity Provider Type Example: idp_type(‘custom’) |
| idp_location | Mandatory | OAuth EndPoint Example: idp_location('http://cnc-tdvm-smp-8878:8181/api/catalog/v1/oauth/tokens')
|
| idp_token_scope | Mandatory for Polaris and Gravitino | OAuth token scope Example: idp_token_scope('PRINCIPAL_ROLE:ALL') |
| storage_location | Mandatory | Object storage path
Example: storage_location ('s3://tdotf-aws-polaris') |
| storage_endpoint | Only for On Prem S3 Compatible Storage and Catalog points to this storage | On Prem S3 Compatible storage end point: Example: storage_endpoint ('https://s3.td.teradata.com:10443' |
| storage_region | Mandatory for AWS S3 or OnPrem S3 Compatible storage like Dell ECS or Minio or StorageGrid | AWS or On Prem Storage Region Example: storage_region ('us-east-1') |