17.10 - Multiple DataConnector Instance Support - Access Module

Teradata® Tools and Utilities Access Module Reference

Product
Access Module
Release Number
17.10
Published
October 2021
Language
English (United States)
Last Update
2021-11-02
dita:mapPath
uur1608578381725.ditamap
dita:ditavalPath
obe1474387269547.ditaval

Teradata Access Module for Kafka supports multiple Dataconnector instances when writing data to Kafka topics.

User can export data from Teradata and then write the data to Kafka topics in a parallel manner using multiple instances of TPT DC operator, resulting in higher throughput.

Teradata Access Module for Kafka distributes the Kafka topic partitions equally to the TPT FileWriter instances which allows exporting data received from DBS to the Kafka topic partitions parallelly.

Example

In this example, we have used 8 FILE_WRITER instances and 16 partitions.

DEFINE JOB EXPORT_FROM_TERADATA
DESCRIPTION 'Export data from Teradata to Kafka topic partitions using Multiple FileWriter instances'
(
      STEP EXPORT_THE_DATA
      (
         APPLY TO OPERATOR ( $FILE_WRITER()[8]
         ATTR
         (
           AccessModuleName = 'libkafkaaxsmod.so',
           AccessModuleInitStr = '-M P -T <TopicName> -B <BrokerID> -P [0-15] -S y -BKSZ 1200'
          )
        )
      SELECT * FROM OPERATOR ( $EXPORT
      ATTR
      (
         SelectStmt = 'SELECT * FROM S3W001_src;'
      )
     );
   );
);

Procedure to Run

tbuild -C -f kafka_multi.tpt -v jobvar_multiinstance -j kafka15
The -C command line option is mandatory for an even distribution of data across the topics. By default, TPT does not distribute the data from the data streams to the file writer instances in a round-robin manner. Use “-C” with tbuild while running multiple Dataconnector instances TPT-Kafka export job.

AccessModuleKafka returns the following error and terminates if the user provides fewer partitions than there are instances:

Dataconnector Instances: %d are greater than total partition count: %d

AccessModuleKafka splits the total 16 partitions and distributes 2 partitions to 8 TPT FileWriter instances equally.

KAFKA_AXSMOD: Running in Multi-Instance mode. This is Instance 8 of 8
KAFKA_AXSMOD: Partition : 1 and Instance number : 8
KAFKA_AXSMOD: Partition : 0 and Instance number : 8

KAFKA_AXSMOD: Running in Multi-Instance mode. This is Instance 3 of 8
KAFKA_AXSMOD: Partition : 11 and Instance number : 3
KAFKA_AXSMOD: Partition : 10 and Instance number : 3

KAFKA_AXSMOD: Running in Multi-Instance mode. This is Instance 6 of 8
KAFKA_AXSMOD: Partition : 5 and Instance number : 6
KAFKA_AXSMOD: Partition : 4 and Instance number : 6

KAFKA_AXSMOD: Running in Multi-Instance mode. This is Instance 2 of 8
KAFKA_AXSMOD: Partition : 13 and Instance number : 2
KAFKA_AXSMOD: Partition : 12 and Instance number : 2

KAFKA_AXSMOD: Running in Multi-Instance mode. This is Instance 4 of 8
KAFKA_AXSMOD: Partition : 9 and Instance number : 4
KAFKA_AXSMOD: Partition : 8 and Instance number : 4

KAFKA_AXSMOD: Running in Multi-Instance mode. This is Instance 5 of 8
KAFKA_AXSMOD: Partition : 7 and Instance number : 5
KAFKA_AXSMOD: Partition : 6 and Instance number : 5

KAFKA_AXSMOD: Running in Multi-Instance mode. This is Instance 7 of 8
KAFKA_AXSMOD: Partition : 3 and Instance number : 7
KAFKA_AXSMOD: Partition : 2 and Instance number : 7

KAFKA_AXSMOD: Running in Multi-Instance mode. This is Instance 1 of 8
KAFKA_AXSMOD: Partition : 15 and Instance number : 1
KAFKA_AXSMOD: Partition : 14 and Instance number : 1