Overview of Teradata Data Mover Components - Teradata Data Mover

Teradata Data Mover User Guide

Product
Teradata Data Mover
Release Number
16.10
Published
June 2017
Language
English (United States)
Last Update
2018-03-29
dita:mapPath
kmo1482331935137.ditamap
dita:ditavalPath
ft:empty
dita:id
B035-4101
lifecycle
previous
Product Category
Analytical Ecosystem
The next figure shows the main components of Data Mover:
  • The Data Mover daemon
  • The Data Mover agent
  • The user interfaces:
    • Graphic (Data Mover portlet)
    • Command-line
    • RESTful API
When you specify job actions using one of the interfaces, the Data Mover daemon generates job execution steps. The Data Mover daemon then distributes the tasks for each job step among the Data Mover agents, which run the job tasks. Data Mover is set up to report job status to Teradata Ecosystem Manager, which can monitor and control Data Mover jobs.
Data Mover Components

Data Mover Deployment Options

Data Mover can be deployed using the following platforms:
  • Teradata Multi-Purpose Server (TMS)
  • Virtual machine on a Consolidated Teradata Multi-Purpose Server (CTMS)
  • IntelliCloud
  • Amazon Web Services
  • Microsoft Azure

Data Mover Daemon

The main component of Teradata Data Mover is the Data Mover daemon, which:

  • Processes incoming requests from the Data Mover command-line interface
  • Queries Teradata Database systems for information on specified objects
  • Generates a plan to copy objects from the source system to the target system, based on job specifications
  • Distributes work for each job to the Data Mover agents
  • Captures and stores job information such as the job plan and output generated by each job instance
  • Sends status information to Teradata Ecosystem Manager and Server Management

The Data Mover daemon uses a scaled-down version of the Teradata Database to store job information and metadata. All communication between the Data Mover daemon and the Data Mover command-line interface, Data Mover agents, or other external components is through the Java Message Service (JMS) bus. All messages between these components are in XML format.

One task of the Data Mover daemon is to send job status information to Teradata Ecosystem Manager and Server Management, enabling critical failures to be reported to Teradata immediately. The Data Mover Daemon generates Teradata Ecosystem Manager event information for jobs launched with the portlet and command-line interfaces, then sends the status to Teradata Ecosystem Manager if that product is present and the user has configured Data Mover to do so.

Properties File for the Data Mover Daemon

The daemon.properties file contains configuration properties for the Data Mover daemon.

# Copyright (C) 2009-2016  by Teradata Corporation.
# All Rights Reserved.
# TERADATA CORPORATION CONFIDENTIAL AND TRADE SECRET
#------------------------------------------------------------
# File: "daemon.properties"
#
# Purpose: This file contains all of the properties used by the  Data Mover  daemon.
#
# Caution: Do not modify the property names/values in this file unless absolutely sure 
# of the implications to the  Data Mover  daemon.
#
# Note: Any line of text in this document that begins with a '#' character is a comment and 
# has no affect on the  Data Mover daemon. However, comments should not be modified.
#
# All properties starting with log4j are used for logging purposes
#
#------------------------------------------------------------

# Purpose: The hostname or IP address of the machine running the 
# Java Message Service (JMS) Message Broker.
# Default: localhost
# Other examples include:
# broker.url=10.0.1.199
# broker.url=hostname
# broker.url=[fe80::114:23ff:fe1d:32fb]
broker.url=localhost

# Purpose: The port number on the machine in which the
# Java Message Service (JMS) Message Broker is listening.
# Default: 61616
broker.port=61616

# Purpose: A long-lived server port on the machine running the Data Mover Daemon that is used for inbound
# socket connections from  Data Mover agents.  
# Default: 25168
arcserver.port=25168

# Purpose: When set to true, Fatal Error messages can be sent to TVI
# Default: true
logger.useTviLogger=true

# Purpose: The maximum number of jobs allowed to run on the daemon at the same time.
# Additional jobs are placed on the queue and executed when slots become available
# Default: 20
jobExecutionCoordinator.maxConcurrentJobs=20

# Purpose: The maximum number of tasks allowed in the job queue.
# Additional jobs are placed in a higher level memory queue until slots are available in the job queue.
# Default: 20
jobExecutionCoordinator.maxQueuedJobs=20

# Purpose: The hostname and IP address for the QueryGrid manager server.
# Could have up to two URLs to support clustered QueryGrid manager, seperated by a comma.
# for example: querygrid.manager.urls=https://servername1:9443,https://servername2:9443
# Default: https://localhost:9443
querygrid.manager.urls=https://localhost::9443

#----------------------LOGGING-------------------------------

# Purpose: Set Logging level to INFO.  User has 6 options.  
# From most amount of logging to least: trace < debug < info < warn < error < fatal
log4j.rootLogger=INFO, logfile

# Purpose: Allows users to change the location of the log file.
# If changing log file location, give the absolute path of the file;
# for example, /var/log/dmAgent.log
# for windows os: use slash instead of back slash: 
# For Example: C:/Program File/Teradata/Log/dmDaemon.log 
log4j.appender.logfile.file=/var/opt/teradata/datamover/logs/dmDaemon.log

# Purpose: Informs the logging application to use a specific appender.  DO NOT CHANGE 
log4j.appender.logfile=org.apache.log4j.RollingFileAppender
# Purpose: The max size of the logging file before being rolled over to backup files.
log4j.appender.logfile.maxFileSize=20MB
# Purpose: The number of backup logging files created, after the max number, the oldest file is erased.
log4j.appender.logfile.maxBackupIndex=5

# Purpose: The Pattern of the layout of log file. d is the date, t is the
# thread, p is the log level, c is the class name and m & n are the messages
# You can create your own layout. For detailed information, go to: 
# http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/PatternLayout.html
log4j.appender.logfile.layout=org.apache.log4j.PatternLayout
log4j.appender.logfile.layout.ConversionPattern=%d [%t] %-5p %c{3}(%L) - %m%n

#------------------------------------------------------------

Data Mover Agent

The Data Mover daemon uses Data Mover agents to perform the work for each Data Mover job; therefore, at least one agent is required. Multiple agents can run tasks in parallel, improving performance. To maximize performance, each agent should reside on its own server, separate from the Data Mover daemon's server. However, it is not a requirement. A Data Mover agent can run on the same server as the Data Mover daemon.

The Data Mover Daemon splits work into tasks, which it places on a JMS Queue for the next available Data Mover agent. The Data Mover agent uses one of the following utilities to perform the task, then listens on the queue for its next task:

  • Teradata Archive/Recovery (ARC)
  • Teradata Parallel Transporter (TPT) API
  • Teradata JDBC Driver
  • QueryGrid
  • Teradata Connector for Hadoop (TDCH)
  • Aster SQL Map-Reduce (MR) Functions

Properties File for the Data Mover Agent

The agent.properties file contains configuration properties for the Data Mover agent.

# Copyright (C) 2009-2016 by Teradata Corporation.
# All Rights Reserved.
# TERADATA CORPORATION CONFIDENTIAL AND TRADE SECRET
#----------------------------------------------------------
# File: "agent.properties"
#
# Purpose: This file contains all of the properties used by the Data Mover agent.
#
# Caution: Do not modify the property names/values in this file unless absolutely sure 
# of the implications to the Data Mover agent.
#
# Note: Any line of text in this document that begins with a '#' character is a comment and 
# has no affect on the  Data Mover  agent. However, comments should not be modified.
#
# All properties starting with log4j are used for logging purposes
#
#---------------------------------------------------------

# Purpose: The Agent Identifier
# Default: Agent1
agent.id=Agent1

# Purpose: The hostname or IP address of the machine running the 
# Java Message Service (JMS) Message Broker.
# Default: localhost
broker.url=localhost

# Purpose: The port number on the machine in which the
# Java Message Service (JMS) Message Broker is listening.
# Default: 61616
broker.port=61616

# Purpose: Port that will be used by ARC
# Default: 25268
arc.port=25268

# Purpose: The maximum number of tasks allowed to run on this agent at the same time.
# This property has the side effect of reducing parallelism among multiple agents if
# set too high, because one agent will grab all the tasks on the queue
# Default: 5
agent.maxConcurrentTasks=5

#When set to true, Fatal Error messages can be sent to TVI
logger.useTviLogger=true

#----------------------LOGGING---------------------------

# Purpose: Set Logging level to INFO.  User has 6 options.  
# From most amount of logging to least: trace < debug < info < warn < error < fatal
log4j.rootLogger=INFO, logfile

# Purpose: Allows users to change the location of the log file.
# If changing log file location, give the absolute path of the file;
# for example, /var/log/dmAgent.log
# for windows os: use slash instead of back slash: 
# For Example: C:/Program File/Teradata/Log/dmAgent.log 
log4j.appender.logfile.file=/var/opt/teradata/datamover/logs/dmAgent.log

# Purpose: Informs the logging application to use a specific appender.  DO NOT CHANGE 
log4j.appender.logfile=org.apache.log4j.RollingFileAppender
# Purpose: The max size of the logging file before being rolled over to backup files.
log4j.appender.logfile.maxFileSize=10MB
# Purpose: The number of backup logging files created, after the max number, the oldest file is erased.
log4j.appender.logfile.maxBackupIndex=3

# Purpose:  The Pattern of the layout of log file.  d is the date, t is the
# thread, p is the log level, c is the class name and m & n are the messages
# You can create your own layout. For detailed information, go to: 
# http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/PatternLayout.html
log4j.appender.logfile.layout=org.apache.log4j.PatternLayout
log4j.appender.logfile.layout.ConversionPattern=%d [%t] %-5p %c{3}(%L) - %m%n

#---------------------------------------------------------

Data Mover Portlet

The Data Mover portlet allows you to set up, launch, edit, and monitor jobs that copy database objects between Teradata Database systems in a graphic-user interface. Because the interface for the Data Mover portlet is user-friendly and intuitive, you can accomplish copy tasks easily with Data Mover by browsing objects to copy in the database hierarchy of the source system and creating a job definition. The portlet eliminates the need to edit XML parameter files or remember the syntax of common commands. To use the Data Mover portlet, you must have an installation of Teradata Viewpoint.

Command-Line Interface

The command-line interface provides setup commands that enable you to set up and configure the Data Mover system, including the daemon and agents and job operation commands that enable you to create, execute, monitor, update, edit, and delete jobs. The setup commands provide functionality that corresponds to that provided by the Data Mover Setup portlet. The job management commands provide functionality that corresponds to that provided by the Data Mover portlet.
For a list of commands and valid parameters, see About Data Mover Commands.
Each command in the interface requires a set of parameters for operation, which can be listed as part of the command-line arguments or specified in an XML document. If the same parameter is defined on the command line and in the parameter file, the command line takes precedence.
Multiple instances of the command-line interface are valid.

RESTful API

The RESTful API provides functionality similar to the command-line interface. The RESTful API can be accessed from any third-party or Teradata server using any standard REST client. The APIs provide a programmatic method to create, execute, monitor, update, edit, stop, clean up, and delete jobs. For more information, see Data Mover RESTful API.