In this setup, catalog access is authenticated using tokens issued by the Databricks Unity OAuth service, which supports Databricks-managed service principals. Storage access is authenticated using Azure IDP credentials.
| Parameter | Description | Requirement | Example Value |
|---|---|---|---|
| catalog_type | Specifies the Iceberg catalog type. Allowed value: rest |
Mandatory | catalog_type ('rest') |
| rest_catalog_type | Specifies REST Iceberg catalog type Allowed value: unity |
Mandatory | rest_catalog_type ('unity') |
| catalog_location | Databricks Unity Iceberg REST API endpoint Format: "<workspace-url>/api/2.1/unity-catalog/iceberg-rest"
This format is according to Databricks Runtime 16.4 LTS and above. |
Mandatory | catalog_location ('https://adb-8074943983864086.6.azuredatabricks.net/api/2.1/unity-catalog/iceberg-rest) |
| catalog_name | Databricks Unity Iceberg Catalog Name | Mandatory | catalog_name ('reg_iceberg_db') |
| idp_type | Identity provider type Allowed values:
If not specified, OTF defaults to idp_type('custom'). Databricks Unity provides a default OAuth service supporting grant_type=client_credentials. |
Optional | idp_type ('custom') |
| idp_location | Specifies the complete URL of the identity provider (IDP) endpoint used to issue access tokens for catalog authentication. This URL is typically created by appending the OAuth token path to the base URL of your OAuth service (for example, the Databricks workspace URL).
This format is according to Databricks Runtime 16.4 LTS and above. |
Optional | idp_location ('https://adb-8074943983864086.6.azuredatabricks.net/oidc/v1/token'') |
| idp_token_scope | Token scope for token generation requests (defaults to 'all-apis' if not provided) | Optional | idp_token_scope ('all-apis') |
| storage_location | AWS Cloud Data Lake Storage location | Mandatory | storage_location |
| tenant_id | AZURE tenant id | Mandatory | tenant_id ('391c8c4c-6a2a-40fd-ab98-226b6baa5155') |
| storage_account_name | AZURE storage account name | Mandatory | storage_account_name ('regicebergstorageacct') |
Example
- Replace azure_databricks_unity_managed_principal_client_id and azure_databricks_unity_managed_principal_secret_key variables with your Databricks OAuth service credentials (Databricks Managed Service Principal).
- Replace azure_user_id and azure_password variables with your Azure IDP credentials.
REPLACE AUTHORIZATION azure_dbx_unity_catalog_auth
USER '<<azure_databricks_unity_managed_principal_client_id>>'
PASSWORD 'azure_databricks_unity_managed_principal_secret_key';
REPLACE AUTHORIZATION azure_idp_auth
USER '<<azure_user_id>>'
PASSWORD '<<azure_password>>';
REPLACE DATALAKE iceberg_unity_azure_dbx_rest
EXTERNAL SECURITY CATALOG azure_dbx_unity_catalog_auth,
EXTERNAL SECURITY STORAGE azure_idp_auth
USING
catalog_type('rest')
rest_catalog_type('unity')
catalog_location('https://adb-8074943983864086.6.azuredatabricks.net/api/2.1/unity-catalog/iceberg-rest')
catalog_name('reg_iceberg_db')
storage_location('abfss://reg-storage-acct-container@regicebergstorageacct.dfs.core.windows.net/9e12180b-1bd5-44bb-ab39-8f70763bb9f5/')
tenant_id('391c8c4c-6a2a-40fd-ab98-226b6baa5155')
storage_account_name('regicebergstorageacct')
TABLE FORMAT iceberg;