- The Data Mover daemon
- The Data Mover agent
- The user interfaces:
- Graphic (Data Mover portlet)
- Command-line
- RESTful API
Data Mover Deployment Options
- Teradata Multipurpose Server (TMS)
- Virtual machine on a Consolidated Teradata Multipurpose Server (CTMS)
- Teradata Cloud (previously IntelliCloud)
- Private Cloud on virtual machine
- Amazon Web Services (AWS)
- Microsoft Azure (Azure)
- Google Cloud
Data Mover Data Transfer Options
From System | To System |
---|---|
VantageCore IntelliFlex | VantageCore IntelliFlex |
VantageCloud Enterprise (public or private) | VantageCloud Enterprise (public or private) |
VantageCore IntelliFlex | VantageCloud Enterprise (public or private) |
VantageCloud Enterprise (public or private) | VantageCore IntelliFlex |
Teradata Cloud | VantageCloud Enterprise (public or private) |
VantageCloud Enterprise (public or private) | Teradata Cloud |
Data Mover Daemon
The main component of Teradata® Data Mover is the Data Mover daemon, which can do the following:
- Processes incoming requests from one of the Data Mover interfaces
- Queries database systems for information on specified objects
- Generates a plan to copy objects from the source system to the target system, based on job specifications
- Distributes work for each job to the Data Mover agents
- Captures and stores job information such as the job plan and output generated by each job instance
- Sends status information to Teradata Ecosystem Manager and Server Management
The Data Mover daemon uses a scaled-down version of Analytics Database to store job information and metadata. All communication between the Data Mover daemon and the Data Mover command-line interface, Data Mover agents, or other external components is through the Java Message Service (JMS) bus. All messages between these components are in XML format.
One task of the Data Mover daemon is to send job status information to Teradata Ecosystem Manager and Server Management, enabling critical failures to be reported to Teradata immediately. The Data Mover Daemon generates Teradata Ecosystem Manager event information for jobs launched with the portlet and command-line interfaces, then sends the status to Teradata Ecosystem Manager if that product is present and the user has configured Data Mover to do so.
Properties File for the Data Mover Daemon
The daemon.properties file contains configuration properties for the Data Mover daemon.
# Copyright (C) 2009-2021 by Teradata Corporation. # All Rights Reserved. # TERADATA CORPORATION CONFIDENTIAL AND TRADE SECRET #---------------------------------------------------------------------------------------------- # File: "daemon.properties" # # Purpose: This file contains all of the properties used by the DM Daemon. # # Caution: Do not modify the property names/values in this file unless absolutely sure # of the implications to the DM Daemon. # # Note: Any line of text in this document that begins with a '#' character is a comment and # has no affect on the DM Daemon. However, comments should not be modified. # # All properties under LOGGING comment are used for logging purposes # #---------------------------------------------------------------------------------------------- # Purpose: The hostname or IP address of the machine running the # Java Message Service (JMS) Message Broker. # Default: localhost # Other examples include: # broker.url=10.0.1.199 # broker.url=hostname # broker.url=[fe80::114:23ff:fe1d:32fb] broker.url=localhost # Purpose: The port number on the machine in which the # Java Message Service (JMS) Message Broker is listening. # Default: 61616 broker.port=61616 # Purpose: When set to true, a connection to a # Secondary Java Message Service (JMS) Message Broker can be # established in case the Primary Java Message Service (JMS) Broker fails. # Default: false cluster.enabled=false # Purpose: The message time to live in milliseconds; zero is unlimited # Be SURE to synchronize clocks of all dm hosts so messages do not expire unintentionally # causing failed or hanging requests to Daemon. # Default: 1 hr jms.response.timetolive=3600000 # Purpose: A long-lived server port on the machine running the DM Daemon, which is used for inbound # socket connections from DM Agents. # Default: 25168 # Purpose: When set to true, Fatal Error messages can be sent to TVI # Default: true tvi.useLogger=true # Purpose: Scripts are used to run to collect TVI diagnostic bundles file. # DO NOT change this directory location. tvi.diagnosticbundle.script.dir=/opt/teradata/datamover/support/diagnosticbundle # Purpose: TVI diagnostic bundle files are saved to this directory # This value need to be the same as SUPPORT_BUNDLE_DIR in /etc/opt/teradata/sm3g. sm3gnode.conf tvi.diagnosticbundle.dir=/var/opt/teradata/datamover/support/diagnosticbundle # Purpose: The maximum number of jobs allowed to run on the daemon at the same time. # Additional jobs are placed on the queue and executed when slots become available # Default: 20 jobExecutionCoordinator.maxConcurrentJobs=20 # Purpose: The maximum number of jobs allowed in the job queue. # Additional jobs are placed in a higher level memory queue until slots are available in the job queue. # Default: 20 jobExecutionCoordinator.maxQueuedJobs=20 # Purpose: The hostname or IP address for the ViewPoint Authentication server. # Default: http://localhost viewpoint.url=http://dm-vp1 # Purpose: The port number for the ViewPoint Authentication server. # Default: 80 viewpoint.port=80 # Purpose: The hostname and IP address for the Querygrid Manager servers. # Support clustered QueryGrid managers, could have upto 2 urls, separate by comma. # e.g querygrid.manager.urls=https://host1:9443,https://host2:9443 # Default: https://localhost:9443 querygrid.manager.urls=https://localhost:9443 #----------------------LOGGING------------------------------- # Purpose: Set Logging level to info. User has 6 options. # From most amount of logging to least: trace < debug < info < warn < error < fatal rootLogger.level=info # Purpose: Informs the logging application to use a specific appender and it's properties. DO NOT CHANGE appender.rolling.type=RollingFile appender.rolling.name=RollingFile appender.rolling.layout.type=PatternLayout appender.rolling.layout.pattern=%d [%t] %-5p %c{3}(%L) - %m%n appender.rolling.policies.type=Policies appender.rolling.policies.size.type=SizeBasedTriggeringPolicy appender.rolling.strategy.type=DefaultRolloverStrategy logger.rolling.name=com.teradata logger.rolling.appenderRef.rolling.ref=RollingFile # Purpose: Allow's user the ability to change the location of the log file. # If changing log file location, please give the absolute path of the file; # for example, /var/log/dmAgent.log # for windows os: use slash instead of back slash: # For Example: C:/Program File/Teradata/Log/dmDaemon.log appender.rolling.fileName=/var/opt/teradata/datamover/logs/dmDaemon.log appender.rolling.filePattern=/var/opt/teradata/datamover/logs/dmDaemon.log.%i # Purpose: The max size of the logging file before being rolled over to backup files. appender.rolling.policies.size.size=20MB # Purpose: The number of backup logging files created, after the max number, the oldest file is erased. appender.rolling.strategy.max=5 service_user=dmuser # Purpose: Data Mover Rest Interface for DSA job status notification dm.rest.endpoint=https://localhost:1443/datamover # Purpose: DSA Rest Interface for DSA job status notification dsa.rest.endpoint=https://localhost:9090/dsa #------------------------------------------------------------
Data Mover Agent
The Data Mover daemon uses Data Mover agents to perform the work for each Data Mover job; therefore, at least one agent is required. Multiple agents can run tasks in parallel, improving performance. To maximize performance, Teradata recommends that each agent resides on its own server, separate from the Data Mover daemon server. However, it is not a requirement. A Data Mover agent can run on the same server as the Data Mover daemon.
The Data Mover daemon splits work into tasks, which are placed on a JMS queue for the next available Data Mover agent. The Data Mover agent uses one of the following utilities to perform the task, then listens on the queue for its next task:
- Data Stream Architecture (DSA)
- Teradata Parallel Transporter (TPT) API
- Teradata JDBC Driver
- QueryGrid
Properties File for the Data Mover Agent
The agent.properties file contains configuration properties for the Data Mover agent.
# Copyright (C) 2009-2021 by Teradata Corporation. # All Rights Reserved. # TERADATA CORPORATION CONFIDENTIAL AND TRADE SECRET #---------------------------------------------------------------------------------------------- # File: "agent.properties" # # Purpose: This file contains all of the properties used by the DM Agent. # # Caution: Do not modify the property names/values in this file unless absolutely sure # of the implications to the DM Agent. # # Note: Any line of text in this document that begins with a '#' character is a comment and # has no affect on the DM Agent. However, comments should not be modified. # # All properties under LOGGING Comment are used for logging purposes # #---------------------------------------------------------------------------------------------- # Purpose: The Agent Identifier # Default: Agent1 agent.id=Agent1 # Purpose: The hostname or IP address of the machine running the # Java Message Service (JMS) Message Broker. # Default: localhost broker.url=localhost # Purpose: The port number on the machine in which the # Java Message Service (JMS) Message Broker is listening. # Default: 61616 broker.port=61616 # Purpose: When set to true, a connection to a # Secondary Java Message Service (JMS) Message Broker can be # established in case the Primary Java Message Service (JMS) Broker fails. # Default: false cluster.enabled=false # Purpose: Port that will be used by ARC # Default: 25268 # Purpose: The maximum number of tasks allowed to run on this agent at the same time. # This property has the side-effect of reducing parallelism among multiple Agents if # set too high, because one Agent will grab all the tasks on the queue # Default: 5 agent.maxConcurrentTasks=5 #When set to true, Fatal Error messages can be sent to TVI tvi.useLogger=true # Purpose: Scripts are used to run to collect TVI diagnostic bundles file. # DO NOT change this directory location. tvi.diagnosticbundle.script.dir=/opt/teradata/datamover/support/diagnosticbundle # Purpose: TVI diagnostic bundle files are saved to this directory # This value need to be the same as SUPPORT_BUNDLE_DIR in /etc/opt/teradata/sm3g. sm3gnode.conf tvi.diagnosticbundle.dir=/var/opt/teradata/datamover/support/diagnosticbundle #----------------------LOGGING------------------------------- # Purpose: Set Logging level to info. User has 6 options. # From most amount of logging to least: trace < debug < info < warn < error < fatal rootLogger.level=info # Purpose: Informs the logging application to use a specific appender and it's properties. DO NOT CHANGE appender.rolling.type=RollingFile appender.rolling.name=RollingFile appender.rolling.layout.type=PatternLayout appender.rolling.layout.pattern=%d [%t] %-5p %c{3}(%L) - %m%n appender.rolling.policies.type=Policies appender.rolling.policies.size.type=SizeBasedTriggeringPolicy appender.rolling.strategy.type=DefaultRolloverStrategy logger.rolling.name=com.teradata logger.rolling.appenderRef.rolling.ref=RollingFile # Purpose: Allow's user the ability to change the location of the log file. # If changing log file location, please give the absolute path of the file; # for example, /var/log/dmAgent.log # for windows os: use slash instead of back slash: # For Example: C:/Program File/Teradata/Log/dmAgent.log appender.rolling.fileName=/var/opt/teradata/datamover/logs/dmAgent.log appender.rolling.filePattern=/var/opt/teradata/datamover/logs/dmAgent.log.%i # Purpose: The max size of the logging file before being rolled over to backup files. appender.rolling.policies.size.size=10MB # Purpose: The number of backup logging files created, after the max number, the oldest file is erased. appender.rolling.strategy.max=3 service_user=dmuser #------------------------------------------------------------
Data Mover Portlet
The Data Mover portlet allows you to set up, launch, edit, and monitor jobs that copy database objects between database systems in a graphic-user interface. Because the interface for the Data Mover portlet is user-friendly and intuitive, you can accomplish copy tasks easily with Data Mover by browsing objects to copy in the database hierarchy of the source system and creating a job definition. The portlet eliminates the need to edit XML parameter files or remember the syntax of common commands. To use the Data Mover portlet, you must have Teradata Viewpoint installed.
Command-Line Interface
The command-line interface provides setup commands that enable you to set up and configure the Data Mover system, including the daemon and agents and job operation commands that enable you to create, run, monitor, update, edit, and delete jobs. The setup commands provide functionality that corresponds to that provided by the Data Mover Setup portlet. The job management commands provide functionality that corresponds to that provided by the Data Mover portlet. For a list of commands and valid parameters, see About Data Mover Commands.
Each command in the interface requires a set of parameters for operation, which can be listed as part of the command-line arguments or specified in an XML document. If the same parameter is defined on the command line and in the XML file, the command line takes precedence. Multiple instances of the command-line interface are valid.
RESTful API
The RESTful API provides functionality similar to the command-line interface. The RESTful API can be accessed from any third-party or Teradata server using any standard REST client. The APIs provide a programmatic method to create, run, monitor, update, edit, stop, clean up, and delete jobs. For more information, see Data Mover RESTful API.