If more than one operation tries to concurrently update the Delta Lake metadata, it could result in the error message, “Metadata of the Delta table has been changed by a concurrent update. Please try the operation again.”
This is an expected behavior when a concurrent update/write is done on the same table. OTF gets an optimistic lock on the table at the start of the write operation; it retrieves the latest version of the metadata at the start of the write operation. Then it goes about the write operation like updating or deleting rows, etc. and eventually, tries to commit that transaction into the table metadata. Prior to that, it checks again if the latest version of the metadata for the table is still the same version it retrieved at the start of the write process. If it is not due to another process updating the metadata while it is processing the write, then it will result in this error message. It does not do a retry automatically because it does not know the transaction that preceded it. Doing so could potentially result in data integrity issues. Therefore, the user must resubmit the query if they still want the update to occur.