17.00 - TPump EcoSystem Manager Integration - Parallel Data Pump

Teradata® Parallel Data Pump Reference

prodname
Parallel Data Pump
vrm_release
17.00
created_date
June 2020
category
Programming Reference
featnum
B035-3021-220K

Teradata EcoSystem Manager (previously Teradata Multi-System Manager, TMSM) monitors and analyzes the relationships of the applications, servers, jobs, tables, and software daemons that comprise it. The centralized interface helps you troubleshoot and determine the impact of any failure in your ecosystem. The expected users of this facility are administrators of the Enterprise Data Warehouse (EDW) such as database administrators, ETL maintenance staff, systems administrators, or anyone who needs to monitor and control processes that include, but are not limited to, Teradata Load and Unload Utilities, Teradata SQL, ETL tools, and Vantage.

To integrate with EcoSystem Manager, TPump has been enhanced to collect operational metadata and event data, obtain the Unit of Work ID (UOW ID) from EcoSystem Manager for a job, and send such data to EcoSystem Manager for monitoring purposes using the “send event” interface as described in the EcoSystem Manager Event System API Reference. By default, a TPump job will send events to EcoSystem Manager as long as EcoSystem Manager is active. If EcoSystem Manager is not active, the job will continue to run, except that no events will be sent to EcoSystem Manager.

Simple ETL process monitoring requires the tracking of the “start” and “end” of the process, which can include multiple “steps”, each of which represents an activity or event of the process to be monitored. In terms of TPump, a load job can be regarded as such a process.

TPump follows the flow required by EcoSystem Manager:

  1. Obtain a system-generated UOW ID from EcoSystem Manager for a TPump job.
  2. Send a “start” event to EcoSystem Manager along with the UOW ID.
  3. Optionally send one or more “step” events to EcoSystem Manager along with the UOW ID.
  4. Send an “end” event to EcoSystem Manager along with the UOW ID.

Below is an example of how TPump job containing two BEGIN EXPORT/END LOAD tasks and two IMPORT tasks sends event messages to EcoSystem Manager in order for the job to be monitored:

At job start: "Connecting session(s)"
Task 1 Import 1: Loading begins
Task 1 Import 1: Checkpoint completes
Task 1 Import 1: Loading completes
Task 1 Import 2: Loading begins
Task 1 Import 2: Checkpoint completes
Task 1 Import 2: Loading completes
Task 2 Import 1: Loading begins
Task 2 Import 1: Checkpoint completes
Task 2 Import 1: Loading completes
Task 2 Import 2: Loading begins
Task 2 Import 2: Checkpoint completes
Task 2 Import 2: Loading completes
At job end: "Job terminating”

“Task n” refers to multiple BEGIN LOAD/END LOAD tasks, “Import n” refers to multiple IMPORTs. TPump sends number of rows inserted/updated/deleted while loading completes.

For detailed information on setting up the EcoSystem Manager environment, refer to the Teradata® Ecosystem Manager Installation, Configuration, and Upgrade Guide for Customers (B035-3203).