OTF can be configured to use Databricks Unity as the Iceberg catalog and AWS Cloud Storage for data. Catalog access is authenticated using tokens issued by the Databricks Unity OAuth service, which supports only Databricks‑managed service principals. Storage access is authenticated using AWS credentials, including an Access Key and a Secret Key.
The configuration parameters are available in the following table:
| Parameter | Description | Requirement | Example Value |
|---|---|---|---|
| catalog_type | Specifies the Iceberg catalog type. Allowed value: rest | Mandatory | catalog_type ('rest') |
| rest_catalog_type | Specifies REST Iceberg catalog type Allowed value: unity |
Mandatory | rest_catalog_type ('unity') |
| catalog_location | Databricks Unity Iceberg REST API endpoint Format: "<workspace-url>/api/2.1/unity-catalog/iceberg-rest"
|
Mandatory | catalog_location ('https://dbc-9f675b6e-febc.cloud.databricks.com/api/2.1/unity-catalog/iceberg-rest') |
| catalog_name | Databricks Unity Iceberg Catalog Name | Mandatory | catalog_name ('vim_iceberg_catalog') |
| idp_type | Identity provider type Allowed values:
If not specified, OTF defaults to idp_type('custom'). Databricks Unity provides a default OAuth service supporting grant_type=client_credentials. |
Optional | idp_type ('custom') |
| idp_location | Specifies the complete URL of the identity provider (IDP) endpoint used to issue access tokens for catalog authentication. This URL is typically created by appending the OAuth token path to the base URL of your OAuth service (for example, the Databricks workspace URL).
|
Optional | idp_location ('https://dbc-9f675b6e-febc.cloud.databricks.com/oidc/v1/token) |
| idp_token_scope | Token scope for token generation requests (defaults to 'all-apis' if not provided) | Optional | idp_token_scope ('all-apis') |
| storage_location | AWS Cloud Storage location | Mandatory | storage_location |
| storage_region | AWS Region | Mandatory | storage_region ('us-west-2') |
Example
- Replace aws_databricks_unity_managed_principal_client_id and aws_databricks_unity_managed_principal_client_secret variables with your Databricks OAuth service credentials (Databricks Managed Service Principal).
- Replace aws_access_key and aws_secret_key variables with your AWS credentials.
REPLACE AUTHORIZATION aws_dbx_unity_catalog_auth
USER '<<aws_databricks_unity_managed_principal_client_id>>'
PASSWORD '<<aws_databricks_unity_managed_principal_secret_key>>';
REPLACE AUTHORIZATION aws_auth_storage
USER '<<aws_access_key>>'
PASSWORD '<<aws_secret_key>>';
REPLACE DATALAKE iceberg_unity_aws_dbx_rest
EXTERNAL SECURITY CATALOG aws_dbx_unity_catalog_auth,
EXTERNAL SECURITY STORAGE aws_auth_storage
USING
catalog_type ('rest')
rest_catalog_type('unity')
catalog_location ('https://dbc-9f675b6e-febc.cloud.databricks.com/api/2.1/unity-catalog/iceberg-rest')
catalog_name ('vim_iceberg_catalog')
storage_location ('s3://vim-databricks/tdotf_metastore_uswest2/ac974248-70e6-4942-8ced-ff7410fabd2e')
storage_region ('us-west-2')
TABLE FORMAT iceberg;
AWS Databricks does not use AWS Managed Service Principals for catalog access. Instead, Databricks Unity Catalog supports only Databricks-managed service principals for OAuth-based catalog authentication. Therefore, for AWS Databricks, the only supported option for accessing the Catalog Service is a Databricks-managed service principal.