VantageCloud Lake uses Data Stream Architecture (DSA) for backup and restores. During backup process, DSA reads the Teradata file system at the data block level and packages them into the savesets on the target media. Various components of DSA run the whole backup operation, which involves:
- Reading data blocks from the database file system.
- Packaging them into buffers that can be transmitted through a network.
- Sending the data packets to the Network Clienthandler, which connects and uploads the data to the eventual target media such as NetBackup or S3 bucket in the form of files.
- It reads data from the target media savesets using Access modules and Network Clienthandlers.
- Once the data is downloaded from the target media/bucket, the Network Clienthandlers send the data buffers to the DSA component running on the Analytics Database nodes for processing.
- DSA then copies/restores these buffers appropriately into the databases/tables/objects.
In VantageCloud Lake, primary clusters use block storage volumes for traditional Teradata databases and table objects. For this block storage data, DSA supports the backup and restores as described earlier. VantageCloud Lake has another storage tier called External object storage, such as an S3 bucket for AWS, for the Object File System (OFS) objects. DSA manages the data protection for the external object storage data without reading the actual object’s data from the external object storage. Instead, it performs the data protection using In-Place Backup mechanism. Since OFS has version tracking capability to restore the OFS object data, DSA uses OFS time-travel mechanism to bring the data to the point that matches the backup version.
In-place Backup
With the in-place backup model, backup of OFS table means protecting the current version of the OFS table at the time of backup on the database system itself and archiving the dictionary data into backup media (S3 Bucket). The backup operation marks the current version of OFS table as protected.
When a particular version of the OFS table is marked as protected, it cannot be deleted by any of the SQL operations, and it cannot be deleted even if that object version’s retention period expires. An OFS table that has protected object versions cannot be dropped. When the backup retention expires or is deleted, DSA unmarks the version of OFS object so that it is no longer protected.
In-place Restore
Backup of the OFS tables involves running the in-place backups and does not involve reading, backing up the actual OFS object’s data from the external object storage. It involves only backing up the corresponding dictionary information for the OFS tables. The dictionary and metadata rows reside in the block storage volume. Therefore, for restore operations, the target database system must have access to the same OFS storage. During the restore operation, DSA loads all the required dictionary data and connects to the same external object storage for the OFS tables.
As part of dictionary data processing, DSA restores all the dictionary tables/rows related to the OFS objects from the backup media, such as an Amazon S3 bucket, and updates the object’s metadata to point to the version that was backed up. After this is complete, the restore process runs an INSERT SELECT AS OF query to create a new version (as restored version), which is a COPY of OFS DATA associated with the version that was marked as protected when the backup was performed.
Restore as COPY
To restore the table into a different database or table name, you can run COPY job. After a COPY job is run on OFS table, the restore process creates a new OFS table on the target database with the same definition as the source table and runs INSERT SELECT AS OF from the source table data to the target table.