In this setup, catalog access is authenticated using tokens issued by the Databricks Unity OAuth service, which supports Databricks-managed service principals. Storage access is handled using a Google Cloud Service Account.
| Parameter | Description | Required | Example Value |
|---|---|---|---|
| catalog_type | Specifies the Iceberg catalog type | Mandatory | catalog_type ('rest') |
| rest_catalog_type | Specifies REST Iceberg catalog type Allowed value: unity |
Mandatory | rest_catalog_type('unity') |
| catalog_location | Databricks Unity Iceberg REST API endpoint Format: "<workspace-url>/api/2.1/unity-catalog/iceberg-rest"
This format is according to Databricks Runtime 16.4 LTS and above. |
Mandatory | catalog_location ('https://1440666423386241.1.gcp.databricks.com/api/2.1/unity-catalog/iceberg-rest') |
| catalog_name | Databricks Unity Iceberg Catalog Name | Mandatory | catalog_name ('reg_iceberg_unity_gcp') |
| idp_type | Identity provider type Allowed values:
If not specified, OTF defaults to idp_type ('custom'). Databricks Unity provides a default OAuth service supporting grant_type=client_credentials. |
Optional | idp_type('custom') |
| idp_location | Specifies the complete URL of the identity provider (IDP) endpoint for issuing access tokens used in catalog authentication. This URL is usually formed by appending the OAuth resource path to your OAuth service base URL (for example,, Databricks workspace URL).
This format is according to Databricks Runtime 16.4 LTS and above. |
Optional | idp_location ('https://1440666423386241.1.gcp.databricks.com/oidc/v1/token') |
| idp_token_scope | Token scope for token generation requests (defaults to 'all-apis' if not provided) | Optional | idp_token_scope('all-apis') |
| storage_location | Google Cloud Storage location | Mandatory | storage_location ('gs://tdotf-dev/58eff1a8-e4a5-44a9-8847-1692a8cb154c') |
| project_id | Google Cloud Service Account project id | Mandatory | project_id ('tc-otf') |
| client_id | Google Cloud Service Account client id | Mandatory | client_id ('110337432395221507455') |
| client_email | Google Cloud Service Account client email | Mandatory | client_email ('sa-tc-otf-000@tc-otf.iam.gserviceaccount.com') |
Example
- Replace gc_databricks_unity_managed_principal_client_id and gcp_databricks_unity_managed_principal_secret_key variables with your Databricks OAuth service credentials (Databricks Managed Service Principal).
- Replace google_cloud_service_account_client_email and google_cloud_service_account_private_key variables with the client email and private key from your Google Cloud Service Account.
REPLACE AUTHORIZATION gcp_dbx_unity_catalog_auth
USER '<<gc_databricks_unity_managed_principal_client_id>>'
PASSWORD 'gcp_databricks_unity_managed_principal_secret_key';
REPLACE AUTHORIZATION gcp_idp_auth
USER '<<google_cloud_service_account_client_email>>'
PASSWORD '<<google_cloud_service_account_private_key>';
REPLACE DATALAKE iceberg_unity_gcp_dbx_rest
EXTERNAL SECURITY CATALOG gcp_dbx_unity_catalog_auth ,
EXTERNAL SECURITY STORAGE gcp_idp_auth
USING
catalog_type ('rest')
rest_catalog_type('unity')
catalog_location ('https://1440666423386241.1.gcp.databricks.com/api/2.1/unity-catalog/iceberg-rest')
catalog_name ('reg_iceberg_unity_gcp')
storage_location ('gs://tdotf-dev/58eff1a8-e4a5-44a9-8847-1692a8cb154c')
project_id ('tc-otf')
client_id ('110337432395221507455')
client_email ('sa-tc-otf-000@tc-otf.iam.gserviceaccount.com')
TABLE FORMAT iceberg;