OTF can be configured to use Polaris as the Iceberg catalog and Azure Data Lake Storage for data. Storage authentication is handled through an Azure Managed Service Principal, while catalog access uses tokens generated by the configured IDP endpoint.
The configuration parameters are available in the following table:
| Parameter | Description | Requirement | Example Value |
|---|---|---|---|
| catalog_type | Specifies the Iceberg catalog type. Allowed value: rest | Mandatory | catalog_type ('rest') |
| rest_catalog_type | Specifies REST Iceberg catalog type. Allowed value: polaris | Mandatory | rest_catalog_type ('polaris') |
| catalog_location | Polaris Iceberg REST API endpoint. Format: protocol://<polaris-service-url>/api/catalog/
|
Mandatory | catalog_location ('http://pe15-tdvm-smp-0042-01:8191/api/catalog/'') |
| catalog_name | The name of the catalog in Polaris Catalog that contains your tables | Mandatory | catalog_name ('tdotf_azure_polaris_catalog') |
| catalog_headers | Polaris Realm Catalog Headers support in OTF REST header information should be provided as a JSON string. This is required to support the Polaris realm concept in OTF. OTF lets you pass realm details as name-value pairs, matching the configuration on the Polaris service. |
Optional (but recommended to use it) | catalog_headers ('{"Polaris-Realm":"tdotf_dev"}') |
| idp_type | Identity provider type Allowed values:
If not specified, OTF defaults to idp_type ('custom'). Apache Polaris provides a default OAuth service supporting grant_type=client_credentials. |
Optional | idp_type ('custom') |
| idp_location | Endpoint of the identity provider (IDP) issuing access token Polaris comes with default OAuth Service:
You can use your own OAuth service (for example, Keycloak) that supports grant_type=client_credentials by configuring it as an external Identity in Polaris. Make sure that the correct OAuth URL endpoint is provided in the OTF datalake configuration. This enables OTF to request tokens from your OAuth provider as needed.
|
Mandatory | idp_location ('http://pe15-tdvm-smp-0042-01:8191/api/catalog/v1/oauth/tokens'') |
| idp_token_scope | Token scope for token generation requests (defaults to 'PRINCIPAL_ROLE:ALL' if not provided) The role names to be in the following format: PRINCIPAL_ROLE:<role name>.
catalog_admin and service_admin are default roles when setting up a realm and RBAC. If you have custom roles configured for your environment, use their corresponding names in this scope.
|
Optional | idp_token_scope ('PRINCIPAL_ROLE:ALL') |
| storage_location | Azure Data Lake Storage location | Mandatory | storage_location |
| tenant_id | Azure tenant id | Mandatory | tenant_id ('391c8c4c-6a2a-40fd-ab98-226b6baa5155') |
| storage_account_name | Azure Storage Account Name | Mandatory | storage_account_name ('regicebergstorageacct') |
Example
Replace client_id and client_secret variables with your OAuth service credentials, and azure_user_id and azure_password variables with your Azure IDP credentials.
REPLACE AUTHORIZATION polaris_catalog_auth
USER '<<client_id>>'
PASSWORD '<<client_secret>>';
REPLACE AUTHORIZATION azure_idp_auth
USER '<<azure_user_id>>'
PASSWORD '<<azure_password>>';
REPLACE DATALAKE iceberg_polaris_azure_rest
EXTERNAL SECURITY CATALOG polaris_catalog_auth,
EXTERNAL SECURITY STORAGE azure_idp_auth
USING
catalog_type ('rest')
rest_catalog_type('polaris')
catalog_location ('http://pe15-tdvm-smp-0042-01:8191/api/catalog/')
catalog_name('tdotf_azure_polaris_catalog')
catalog_headers('{"Polaris-Realm":"tdotf_dev"}')
idp_location('http://pe15-tdvm-smp-0042-01:8191/api/catalog/v1/oauth/tokens')
storage_location('abfss://reg-storage-acct-container@regicebergstorageacct.dfs.core.windows.net/azure_polaris_catalog/')
tenant_id('391c8c4c-6a2a-40fd-ab98-226b6baa5155')
storage_account_name('regicebergstorageacct')
TABLE FORMAT iceberg;