A Databricks Spark cluster needs to be set up if the Teradata OTF performs Write operations on the Iceberg tables in Unity.
Spark Cluster for Iceberg Write Operations
Once Delta Lake Write has completed its tasks and Iceberg Write has committed the metadata (for some operations, not all), the Iceberg table changes should be complete. Unfortunately, the Iceberg metadata update is not known to Unity catalog since the Iceberg REST catalog is not used. As a managed table, it appears Databricks maintains its own metadata layer and does not read the version of the tables inside the table metadata. Thus, a programmatic trigger of the Iceberg metadata generation from the latest Delta Lake table definition, is required. This action not only generates Iceberg metadata but also updates the Unity catalog of the preceding changes. This is accomplished by Teradata OTF running a Spark SQL programmatically using the Sparks cluster in Databricks:
MSCK REPAIR TABLE <full table name> SYNC METADATA;
The Universal Format is documented in the following location: https://docs.databricks.com/en/delta/uniform.html.
Because of this, Iceberg Write has a dependency on Databricks Spark cluster. Currently this is the only mechanism to update Unity catalog of the Delta Lake/Iceberg Uniform changes since there are no APIs - both REST and in Databricks SDK - to refresh/update a table or update the Unity catalog of the underlying changes to the Delta Lake/Iceberg Uniform tables.