Overview of Components | Teradata Data Mover - Overview of Teradata Data Mover Components - Teradata Data Mover

Teradata® Data Mover User Guide

Product
Teradata Data Mover
Release Number
16.20
Published
November 2021
Language
English (United States)
Last Update
2021-11-04
dita:mapPath
wph1512683331242.ditamap
dita:ditavalPath
4101_UG_dm_1620.ditaval
dita:id
B035-4101
lifecycle
previous
Product Category
Analytical Ecosystem
The next figure shows the main components of Data Mover:
  • The Data Mover daemon
  • The Data Mover agent
  • The user interfaces:
    • Graphic (Data Mover portlet)
    • Command line
    • RESTful API
When specifying job actions using one of the interfaces, the Data Mover daemon generates job steps. The Data Mover daemon then distributes the tasks for each job step among the Data Mover agents, which run the job tasks. Data Mover is set up to report job status to Teradata Ecosystem Manager, which can monitor and control Data Mover jobs.
Data Mover Components
Data Mover components, displaying the architecture and flow of data movement among supported interfaces.

Data Mover Deployment Options

Data Mover can be deployed using the following platforms:
  • Teradata Multi-Purpose Server (TMS)
  • Virtual machine on a Consolidated Teradata Multi-Purpose Server (CTMS)
  • Teradata IntelliCloud
  • Amazon Web Services (AWS)
  • Microsoft Azure (Azure)

Data Mover Data Transfer Options

Data Mover supports data transfer between the following systems:
From System To System
Teradata On-Premises Teradata On-Premises
Teradata On-Premises Hadoop On-Premises
Hadoop On-Premises Teradata On-Premises
Teradata On-Premises Aster On-Premises
Aster On-Premises Teradata On-Premises
Teradata in the Public Cloud (AWS and Azure) Teradata in the Public Cloud (AWS and Azure)
Teradata On-Premises Teradata in the Public Cloud (AWS and Azure)
Teradata in the Public Cloud (AWS and Azure) Teradata On-Premises
Hadoop On-Premises Teradata in the Public Cloud (AWS and Azure)
Teradata in the Public Cloud (AWS and Azure) Hadoop On-Premises
Aster On-Premises Teradata in the Public Cloud (AWS and Azure)
Teradata in the Public Cloud (AWS and Azure) Aster On-Premises
Teradata IntelliCloud Teradata in the Public Cloud (AWS and Azure)
Teradata in the Public Cloud (AWS and Azure) Teradata IntelliCloud

Data Mover Daemon

The main component of Teradata® Data Mover is the Data Mover daemon, which can do the following:

  • Processes incoming requests from one of the Data Mover interfaces
  • Queries Teradata Database systems for information on specified objects
  • Generates a plan to copy objects from the source system to the target system, based on job specifications
  • Distributes work for each job to the Data Mover agents
  • Captures and stores job information such as the job plan and output generated by each job instance
  • Sends status information to Teradata Ecosystem Manager and Server Management

The Data Mover daemon uses a scaled-down version of the Teradata Database to store job information and metadata. All communication between the Data Mover daemon and the Data Mover command-line interface, Data Mover agents, or other external components is through the Java Message Service (JMS) bus. All messages between these components are in XML format.

One task of the Data Mover daemon is to send job status information to Teradata Ecosystem Manager and Server Management, enabling critical failures to be reported to Teradata immediately. The Data Mover Daemon generates Teradata Ecosystem Manager event information for jobs launched with the portlet and command-line interfaces, then sends the status to Teradata Ecosystem Manager if that product is present and the user has configured Data Mover to do so.

Properties File for the Data Mover Daemon

The daemon.properties file contains configuration properties for the Data Mover daemon.

# Copyright (C) 2009-2019  by Teradata Corporation.
# All Rights Reserved.
# TERADATA CORPORATION CONFIDENTIAL AND TRADE SECRET
#----------------------------------------------------------------------------------------------
# File: "daemon.properties"
#
# Purpose: This file contains all of the properties used by the DM Daemon.
#
# Caution: Do not modify the property names/values in this file unless absolutely sure 
# of the implications to the DM Daemon.
#
# Note: Any line of text in this document that begins with a '#' character is a comment and 
# has no affect on the DM Daemon. However, comments should not be modified.
#
# All properties under LOGGING comment are used for logging purposes
#
#----------------------------------------------------------------------------------------------

# Purpose: The hostname or IP address of the machine running the 
# Java Message Service (JMS) Message Broker.
# Default: localhost
# Other examples include:
# broker.url=10.0.1.199
# broker.url=hostname
# broker.url=[fe80::114:23ff:fe1d:32fb]
broker.url=localhost

# Purpose: The port number on the machine in which the
# Java Message Service (JMS) Message Broker is listening.
# Default: 61616
broker.port=61616

# Purpose: When set to true, a connection to a 
# Secondary Java Message Service (JMS) Message Broker can be
# established in case the Primary Java Message Service (JMS) Broker fails.
# Default: false
cluster.enabled=false

# Purpose: The message time to live in milliseconds; zero is unlimited
# Be SURE to synchronize clocks of all dm hosts so messages do not expire unintentionally
# causing failed or hanging requests to Daemon.
# Default: 1 hr
jms.response.timetolive=3600000

# Purpose: A long-lived server port on the machine running the DM Daemon, which is used for inbound
# socket connections from DM Agents.  
# Default: 25168
arcserver.port=25168

# Purpose: When set to true, Fatal Error messages can be sent to TVI
# Default: true
tvi.useLogger=true

# Purpose: Scripts are used to run to collect TVI diagnostic bundles file. 
# DO NOT change this directory location.
tvi.diagnosticbundle.script.dir=/opt/teradata/datamover/support/diagnosticbundle

# Purpose: TVI diagnostic bundle files are saved to this directory
# This value need to be the same as SUPPORT_BUNDLE_DIR in /etc/opt/teradata/sm3g. sm3gnode.conf 
tvi.diagnosticbundle.dir=/var/opt/teradata/datamover/support/diagnosticbundle

# Purpose: The maximum number of jobs allowed to run on the daemon at the same time.
# Additional jobs are placed on the queue and executed when slots become available
# Default: 20
jobExecutionCoordinator.maxConcurrentJobs=20

# Purpose: The maximum number of jobs allowed in the job queue.
# Additional jobs are placed in a higher level memory queue until slots are available in the job queue.
# Default: 20
jobExecutionCoordinator.maxQueuedJobs=20

# Purpose: The hostname or IP address for the ViewPoint Authentication server.
# Default: http://localhost
viewpoint.url=http://dm-vp1

# Purpose: The port number for the ViewPoint Authentication server.
# Default: 80
viewpoint.port=80

# Purpose: The hostname and IP address for the Querygrid Manager servers.
# Support clustered QueryGrid managers, could have upto 2 urls, separate by comma.
# e.g querygrid.manager.urls=https://host1:9443,https://host2:9443
# Default: https://localhost:9443
querygrid.manager.urls=https://localhost:9443

#----------------------LOGGING-------------------------------

# Purpose: Set Logging level to info.  User has 6 options.  
# From most amount of logging to least: trace < debug < info < warn < error < fatal
rootLogger.level=info

# Purpose: Informs the logging application to use a specific appender and it's properties.  DO NOT CHANGE 
appender.rolling.type=RollingFile
appender.rolling.name=RollingFile
appender.rolling.layout.type=PatternLayout
appender.rolling.layout.pattern=%d [%t] %-5p %c{3}(%L) - %m%n
appender.rolling.policies.type=Policies
appender.rolling.policies.size.type=SizeBasedTriggeringPolicy
appender.rolling.strategy.type=DefaultRolloverStrategy
logger.rolling.name=com.teradata
logger.rolling.appenderRef.rolling.ref=RollingFile

# Purpose: Allow's user the ability to change the location of the log file.
# If changing log file location, please give the absolute path of the file;
# for example, /var/log/dmAgent.log
# for windows os: use slash instead of back slash: 
# For Example: C:/Program File/Teradata/Log/dmDaemon.log 
appender.rolling.fileName=/var/opt/teradata/datamover/logs/dmDaemon.log
appender.rolling.filePattern=/var/opt/teradata/datamover/logs/dmDaemon.log.%i

# Purpose: The max size of the logging file before being rolled over to backup files.
appender.rolling.policies.size.size=20MB
# Purpose: The number of backup logging files created, after the max number, the oldest file is erased.
appender.rolling.strategy.max=5

service_user=dmuser

# Purpose: Data Mover Rest Interface for DSA job status notification
dm.rest.endpoint=https://localhost:1443/datamover

# Purpose: DSA Rest Interface for DSA job status notification 
dsa.rest.endpoint=https://localhost:9090/dsa
#------------------------------------------------------------

Data Mover Agent

The Data Mover daemon uses Data Mover agents to perform the work for each Data Mover job; therefore, at least one agent is required. Multiple agents can run tasks in parallel, improving performance. To maximize performance, Teradata recommends that each agent resides on its own server, separate from the Data Mover daemon server. However, it is not a requirement. A Data Mover agent can run on the same server as the Data Mover daemon.

The Data Mover daemon splits work into tasks, which are placed on a JMS queue for the next available Data Mover agent. The Data Mover agent uses one of the following utilities to perform the task, then listens on the queue for its next task:

  • Teradata Archive/Recovery (ARC)
  • Data Stream Architecture (DSA)
  • Teradata Parallel Transporter (TPT) API
  • Teradata JDBC Driver
  • QueryGrid
  • Teradata Connector for Hadoop (TDCH)
  • Aster SQL Map-Reduce (MR) Functions

Properties File for the Data Mover Agent

The agent.properties file contains configuration properties for the Data Mover agent.

# Copyright (C) 2009-2019 by Teradata Corporation.
# All Rights Reserved.
# TERADATA CORPORATION CONFIDENTIAL AND TRADE SECRET
#----------------------------------------------------------------------------------------------
# File: "agent.properties"
#
# Purpose: This file contains all of the properties used by the DM Agent.
#
# Caution: Do not modify the property names/values in this file unless absolutely sure 
# of the implications to the DM Agent.
#
# Note: Any line of text in this document that begins with a '#' character is a comment and 
# has no affect on the DM Agent. However, comments should not be modified.
#
# All properties under LOGGING Comment are used for logging purposes
#
#----------------------------------------------------------------------------------------------

# Purpose: The Agent Identifier
# Default: Agent1
agent.id=Agent1

# Purpose: The hostname or IP address of the machine running the 
# Java Message Service (JMS) Message Broker.
# Default: localhost
broker.url=localhost

# Purpose: The port number on the machine in which the
# Java Message Service (JMS) Message Broker is listening.
# Default: 61616
broker.port=61616

# Purpose: When set to true, a connection to a 
# Secondary Java Message Service (JMS) Message Broker can be
# established in case the Primary Java Message Service (JMS) Broker fails.
# Default: false
cluster.enabled=false

# Purpose: Port that will be used by ARC
# Default: 25268
arc.port=25268

# Purpose: Aster performance tracker execution frequency used by Aster task progress.
# Default: 300 seconds (5 mins), -1 is to disable the progress tracker.
aster.progressTrackerFrequency=300

# Purpose: The maximum number of tasks allowed to run on this agent at the same time.
# This property has the side-effect of reducing parallelism among multiple Agents if
# set too high, because one Agent will grab all the tasks on the queue
# Default: 5
agent.maxConcurrentTasks=5

#When set to true, Fatal Error messages can be sent to TVI
tvi.useLogger=true

# Purpose: Scripts are used to run to collect TVI diagnostic bundles file. 
# DO NOT change this directory location.
tvi.diagnosticbundle.script.dir=/opt/teradata/datamover/support/diagnosticbundle

# Purpose: TVI diagnostic bundle files are saved to this directory
# This value need to be the same as SUPPORT_BUNDLE_DIR in /etc/opt/teradata/sm3g. sm3gnode.conf 
tvi.diagnosticbundle.dir=/var/opt/teradata/datamover/support/diagnosticbundle

#----------------------LOGGING-------------------------------

# Purpose: Set Logging level to info.  User has 6 options.  
# From most amount of logging to least: trace < debug < info < warn < error < fatal
rootLogger.level=info

# Purpose: Informs the logging application to use a specific appender and it's properties.  DO NOT CHANGE 
appender.rolling.type=RollingFile
appender.rolling.name=RollingFile
appender.rolling.layout.type=PatternLayout
appender.rolling.layout.pattern=%d [%t] %-5p %c{3}(%L) - %m%n
appender.rolling.policies.type=Policies
appender.rolling.policies.size.type=SizeBasedTriggeringPolicy
appender.rolling.strategy.type=DefaultRolloverStrategy
logger.rolling.name=com.teradata
logger.rolling.appenderRef.rolling.ref=RollingFile

# Purpose: Allow's user the ability to change the location of the log file.
# If changing log file location, please give the absolute path of the file;
# for example, /var/log/dmAgent.log
# for windows os: use slash instead of back slash: 
# For Example: C:/Program File/Teradata/Log/dmAgent.log 
appender.rolling.fileName=/var/opt/teradata/datamover/logs/dmAgent.log
appender.rolling.filePattern=/var/opt/teradata/datamover/logs/dmAgent.log.%i

# Purpose: The max size of the logging file before being rolled over to backup files.
appender.rolling.policies.size.size=10MB
# Purpose: The number of backup logging files created, after the max number, the oldest file is erased.
appender.rolling.strategy.max=3

service_user=dmuser
#------------------------------------------------------------

Data Mover Portlet

The Data Mover portlet allows you to set up, launch, edit, and monitor jobs that copy database objects between Teradata Database systems in a graphic-user interface. Because the interface for the Data Mover portlet is user-friendly and intuitive, you can accomplish copy tasks easily with Data Mover by browsing objects to copy in the database hierarchy of the source system and creating a job definition. The portlet eliminates the need to edit XML parameter files or remember the syntax of common commands. To use the Data Mover portlet, you must have Teradata Viewpoint installed.

Command-Line Interface

The command-line interface provides setup commands that enable you to set up and configure the Data Mover system, including the daemon and agents and job operation commands that enable you to create, run, monitor, update, edit, and delete jobs. The setup commands provide functionality that corresponds to that provided by the Data Mover Setup portlet. The job management commands provide functionality that corresponds to that provided by the Data Mover portlet. For a list of commands and valid parameters, see About Data Mover Commands.

Each command in the interface requires a set of parameters for operation, which can be listed as part of the command-line arguments or specified in an XML document. If the same parameter is defined on the command line and in the XML file, the command line takes precedence. Multiple instances of the command-line interface are valid.

RESTful API

The RESTful API provides functionality similar to the command-line interface. The RESTful API can be accessed from any third-party or Teradata server using any standard REST client. The APIs provide a programmatic method to create, run, monitor, update, edit, stop, clean up, and delete jobs. For more information, see Data Mover RESTful API.