OTF can be configured to use Polaris as the Iceberg catalog and Google Cloud Storage for data. Storage access is handled through a Google Service Account, while catalog access is authenticated using tokens generated by the configured IDP endpoint.
The configuration parameters are available in the following table:
| Parameter | Description | Requirement | Example Value |
|---|---|---|---|
| catalog_type | Specifies the Iceberg catalog type. Allowed value: rest | Mandatory | catalog_type ('rest') |
| rest_catalog_type | Specifies REST Iceberg catalog type. Allowed value: polaris | Mandatory | rest_catalog_type ('polaris') |
| catalog_location | Polaris Iceberg REST API endpoint. Format: protocol://<polaris-service-url>/api/catalog/
|
Mandatory | catalog_location ('http://pe15-tdvm-smp-0042-01:8191/api/catalog/'') |
| catalog_name | Polaris Iceberg Catalog Name | Mandatory | catalog_name ('tdotf_gcp_polaris_catalog') |
| catalog_headers | Polaris Realm Catalog Headers support in OTF REST header information should be provided as a JSON string. This is required to support the Polaris realm concept in OTF. OTF lets you pass realm details as name-value pairs, matching the configuration on the Polaris service. |
Optional (but recommended to use it) | catalog_headers ('{"Polaris-Realm":"tdotf_dev"}') |
| idp_type | Identity provider type Allowed values:
If not specified, OTF defaults to idp_type ('custom'). Apache Polaris provides a default OAuth service supporting grant_type=client_credentials. |
Optional | idp_type('custom') |
| idp_location | Endpoint of the identity provider (IDP) issuing access token Polaris comes with default OAuth Service:
You can use your own OAuth service (for example, Keycloak) that supports grant_type=client_credentials by configuring it as an external Identity in Polaris. Make sure that the correct OAuth URL endpoint is provided in the OTF datalake configuration. This enables OTF to request tokens from your OAuth provider as needed.
|
Mandatory | idp_location ('http://pe15-tdvm-smp-0042-01:8191/api/catalog/v1/oauth/tokens'') |
| idp_token_scope | Token scope for token generation requests (defaults to 'PRINCIPAL_ROLE:ALL' if not provided) The role names to be in the following format: PRINCIPAL_ROLE:<role name>.
catalog_admin and service_admin are default roles when setting up a realm and RBAC. If you have custom roles configured for your environment, use their corresponding names in this scope.
|
Optional | idp_token_scope ('PRINCIPAL_ROLE:ALL') |
| storage_location | Google Cloud Storage location | Mandatory | storage_location ('gs://tdotf-gcp-polaris/') |
| project_id | Google Cloud Service Account project id | Mandatory | project_id ('tc-otf') |
| client_id | Google Cloud Service Account project id | Mandatory | client_id ('110337432395221507455') |
| client_email | Google Cloud Service Account client email | Mandatory | client_email ('sa-tc-otf-000@tc-otf.iam.gserviceaccount.com') |
Example
- Replace client_id and client_secret variables with your OAuth service credentials.
- Replace google_cloud_service_account_client_email and google_cloud_service_account_private_key variables with the client email and private key from your Google Cloud Service Account.
REPLACE AUTHORIZATION polaris_catalog_auth
USER '<<client_id>>'
PASSWORD '<<client_secret>>';
REPLACE AUTHORIZATION gcp_idp_auth
USER '<<google_cloud_service_account_client_email>>'
PASSWORD '<<google_cloud_service_account_private_key>';
REPLACE DATALAKE iceberg_polaris_gcp_rest
EXTERNAL SECURITY CATALOG polaris_catalog_auth,
EXTERNAL SECURITY STORAGE gcp_idp_auth
USING
catalog_type ('rest')
rest_catalog_type('polaris')
catalog_location ('http://pe15-tdvm-smp-0042-01:8191/api/catalog/')
catalog_name('tdotf_gcp_polaris_catalog')
catalog_headers('{"Polaris-Realm":"tdotf_dev"}')
idp_location('http://pe15-tdvm-smp-0042-01:8191/api/catalog/v1/oauth/tokens')
storage_location ('gs://tdotf-gcp-polaris/')
project_id ('tc-otf')
client_id ('110337432395221507455')
client_email ('sa-tc-otf-000@tc-otf.iam.gserviceaccount.com')
TABLE FORMAT iceberg;