17.10 - Down AMP Recovery - Advanced SQL Engine - Teradata Database

Teradata Vantageā„¢ - Database Utilities

Advanced SQL Engine
Teradata Database
Release Number
July 2021
English (United States)
Last Update

Down AMP Recovery is a process that handles all changes to entire tables or rows, either fallback and primary, while the AMP is down or offline. Down AMP Recovery updates a recovering AMP with data that the Teradata system processed while the AMP was down. The down AMP is considered to be in offline catchup mode. Catchup mode indicates that the AMP is logically offline and in the process of updating its tables so that they are synchronized with the online AMPs in the cluster.

In a crash or restart condition, it is possible to lose an AMP from, for example, a CPU board failure without losing its underlying disk data. The remainder of the Teradata system can then restart and recover the transactions without the failed AMP.

After the down AMP is ready to rejoin the Teradata system, the down AMP recovers the lost data by performing the following steps:

  1. Restoring the data to a consistent state, relative to the transactions the down AMP was working on at the time of the failure.

    The down AMP applies the information in its Transient Journal against its underlying data. Moreover, the down AMP must concur with the choice of the rest of the Teradata system whether to roll back or commit each transaction.

  2. Updating the restored data to match all changes made to the online Teradata system while the AMP was down (Down AMP Recovery).

Down AMP Recovery Operations Display

The following display indicates the fields that are referenced in the down AMP recovery operations explanation on the next page. This display is printed out by rcvmanager. For a description of the fields in the following display, see Down/Catchup AMP Recovery Status.


AMP to be           Current Pass       Next Pass
caught up   Pass     OJ    CJ          OJ   CJ
----------  ----     ---   ---------   ---  --------
    0001    0        0     0           1    1,081
    - AMP Status: Online Catchup
    - Not currently executing recovery

    0002    1        5     145,822     1    25,081
    - AMP Status: Offline Catchup
    - Transaction Recovery: 25,488 TJ Rows

    0003    1        0     2,142       0    0
    - AMP Status: Online Catchup
    - Change Row Recovery: 26% complete in this pass 
    0004    2        2     201,558     0    4,228
    - AMP Status: Offline Catchup
    - Rebuilding Database1.Table1: 45% complete

    0005*   4        0     0           0    2,888
    - AMP Status: Offline Catchup
    - Between passes
* - would probably be placed in online catch up if a restart occurred

Recovering Down AMPs

Perform steps one through four only when offline catchup AMPs exist. If an AMP is still physically down, do not perform these steps.

To recover a down AMP, do the following:

  1. Previously down AMPs that are now available begin their local transaction recovery (referred to as offline catchup AMPs). This step must be completed before going to step 2.
  2. The extracted Ordered System Change Journal (OJ) is processed, rebuilding various subtables.
  3. The extracted Changed Row Journal (CJ) rows are then processed by sending the changed rows (or notification that the rows were deleted) to the catchup AMP.
  4. The OJ entries are applied. Ones which take a significant amount of time are the build operations.
  5. Current CJ entries are applied. Each CJ entry represents one row to be updated (insert, delete, update).
  6. Online AMPs in the down AMP cluster extract all the current build records from the OJ and CJ, and sort them to eliminate duplicates. Any CJ entries referring to rows in a table which will be rebuilt are deleted. Other OJ operations are sent to the catchup AMP for execution.
  7. The Next Pass OJ and CJ entries become the Current Pass entries.
  8. Repeat steps 2 through 7 indefinitely.
  9. After the AMP is sufficiently caught up, it becomes eligible to become an online catchup AMP on the next restart. This is denoted by displaying an asterisk (*) under the column entitled, AMP to be caught up, in the DOWN AMP RECOVERY STATUS screen of the rcvmanager status display. See the previous display screen.

Recovery Journal

When an AMP in a cluster goes down, and the fallback option is specified for a table, the Down AMP Recovery Journal (RJ) records changes made to the fallback tables that are applicable to the down AMP. The journal is active only during an AMP failure and is only used for fallback tables.

The Recovery Journal process for a down AMP is as follows:

  1. Operational AMPs in the cluster begin logging entries into the Down AMP RJ.
  2. The Down AMP RJ records the changes that should have been made to fallback-protected rows of the inoperative AMP.
  3. The changes are applied when the down AMP recovers.

The Recovery Journal maintains two sets of records: the Changed Row Journal (CJ) and the Ordered System Change Journal (OJ).

Changed Row Journal

The CJ logs changed rows in an AMP cluster by logging pointers to the rows that have been changed, but does not log the actual rows. The CJ recovery needs to match its data against all changes made to the online Teradata system while the AMP was down. The most common type of changes includes modifications made to individual rows by Data Manipulation Language (DML) operations of inserting, deleting, and updating rows in pre-existing tables.

Each modified row is remembered in the Teradata system CJ, which is local to each AMP on which the modification takes place. Since the modification is done only while an AMP is down, the CJ is only populated on AMPs in a cluster while an AMP is down. AMPs are arranged into a group called clusters so that each AMP provides fallback protection to other members within that same group.

Entries stored in the CJ include only the table ID and row ID of the row which was modified. When the down AMP recovers, the actual row is extracted from the fallback or primary subtable; the row image is not stored redundantly in the CJ.

Ordered System Change Journal

The OJ logs data changes that are applied to the online Teradata system while an AMP is down, such as building the index, creating a permanent journal, and performing a table rebuild. These types of changes may be termed Teradata system-level changes and involve Data Definition Language (DDL) operations, since they affect all AMPs, not only a single row update.

These changes cover DDL operations such as, DROP TABLE, CREATE TABLE, and CREATE/DROP INDEX. Other operations include a change to every single row within a table, for example, drop a column. For these types of operations, recovery involves copying every single row of the table, for those rows the AMP owns, over to that down AMP (a table rebuild).

Therefore, the majority of the rows found in the OJ are build records that identify tables that need to be built for created tables, or rebuilt when the AMP is recovered. Other OJ records are HUT Lock set and release records, and in-doubt two-phase commit transaction.

Deferred Down AMP Recovery

The process of Deferred Down AMP Recovery means that while a down AMP remains down and recovering, the rest of the Teradata system continues its operations with the connected hosts.

Offline Catchup Mode

In offline catchup mode, the new transactions for the down AMP are performed by other AMPs in the cluster. Therefore, new change row and Teradata system change entries are made while the down AMP is processing the old ones. While the AMP is in catchup mode, the AMP is considered logically offline.

To set a down AMP to offline catchup mode, do the following:

  1. Start the Vproc Manager utility.
  2. To list the Teradata system logical configuration, type:


    A screen similar to the following appears:

    Vproc   Rel.   Node          Crash   Vproc   Config   Config  Cluster/  RvcJrnl/
    Number  Vproc# ID   Movable  Count   State   Status   Type    Host No.   Host Type
    0*      1      0-0  No       0       Online  Online   AMP       0           On
    1       2      0-0  No       0       Offline Down     AMP       0           On
    16383   3      0-0  No       0       Online  Online   PE        52          COP
    * DBS Control AMP
    DBS State: Logons are enabled - Users are logged on
    DBS RestartKind: COLD
    The disable list is empty
  3. To bring up the downed AMP, type:

    set 1 = online;

    If vproc 1 goes into catchup mode, the following message appears:

    Vproc 1 will begin recovery in the background via the Recovery Control Task
  4. To verify that the AMP is in Utility Catchup mode, type:


    While in the offline mode, the AMP has OJ build records to process and/or a large number of CJ rows. This reduces the amount of data that is locked, but any new transactions on the online Teradata system creates additional OJ and CJ rows. Since you cannot place an offline AMP into the online Teradata system, these passes could continue forever or until the Teradata system restarts.

    A screen similar to the following appears:

    Vproc   Rel.   Node          Crash   Vproc   Config   Config  Cluster/  RvcJrnl/
    Number  Vproc# ID   Movable  Count   State   Status   Type    Host No.   Host Type
    0*      1      0-0  No       0       Online  Online   AMP       0           On
    1       2      0-0  No       0       Utility Catchup  AMP       0           On
    16383   3      0-0  No       0       Online  Online   PE        52          COP
    * DBS Control AMP
    DBS State: Logons are enabled - The system is quiescent
    DBS RestartKind: COLD
    The disable list is empty

To verify if an offline AMP, for example, is almost in catchup, do the following:

  1. Start rcvmanager.
  2. To check the offline AMP status, type:

    list status;

    One of the following messages appears:

    Message... Means that the...
    - AMP Status: Not in recovery

    - Down

    AMP is still down.
    - AMP Status: Offline Catchup

    - Executing miscellaneous Recovery actions

    AMP is in catchup mode.
    - AMP Status: Offline Catchup

    - Between Passes

    * - would probably be placed in online catchup if a restart occurs.

    AMP is in catchup mode, and if a restart occurs, the AMP is placed in online catchup mode.

If a COLDWAIT restart is performed, the operations are similar to those of Transaction Recovery. In COLDWAIT, the Teradata system remains offline with no incoming transactions until all recovering AMPs have fully recovered. After recovery, the AMPs are all set to online status, and the Teradata system completes start-up.

In offline catchup, catchup tries to run faster than the new update transactions coming in online. The other AMPs handle the fallback responsibility for the down AMP, as well as the additional work involved in writing CJ or OJ records.

If an AMP is designated to be in offline catchup mode, then you must initiate a COLD restart, but only after the AMP has sufficiently recovered, to bring the offline AMP back online. "Sufficiently recovered" refers to the third message in the table above.

Online Catchup Mode

In online catchup mode, the previously down AMP will also accept transactions, as will the rest of the Teradata system. In this mode, all the data that needs updating is locked, so that new data does not operate on the obsolete data.

When the down AMP has a small amount to recover, it can be placed into online catchup.

Since the AMP is online and participating in the new transaction coming into the Teradata system, no new CJ or OJ entries are being created for it. After all of the residual CJ processing is completed, the Teradata system becomes online. Start the Vproc Manager utility and check the status of the database. The Vproc State and Config Status should appear as Online. At this point, the AMP rejoins the other online AMPs automatically without the need of a Teradata system restart.