Teradata Access Module for Kafka supports multiple Dataconnector instances when writing data to Kafka topics.
User can export data from Teradata and then write the data to Kafka topics in a parallel manner using multiple instances of TPT DC operator, resulting in higher throughput.
Teradata Access Module for Kafka distributes the Kafka topic partitions equally to the TPT FileWriter instances which allows exporting data received from DBS to the Kafka topic partitions parallelly.
Example
In this example, we have used 8 FILE_WRITER instances and 16 partitions.
DEFINE JOB EXPORT_FROM_TERADATA
DESCRIPTION 'Export data from Teradata to Kafka topic partitions using Multiple FileWriter instances'
(
STEP EXPORT_THE_DATA
(
APPLY TO OPERATOR ( $FILE_WRITER()[8]
ATTR
(
AccessModuleName = 'libkafkaaxsmod.so',
AccessModuleInitStr = '-M P -T <TopicName> -B <BrokerID> -P [0-15] -S y -BKSZ 1200'
)
)
SELECT * FROM OPERATOR ( $EXPORT
ATTR
(
SelectStmt = 'SELECT * FROM S3W001_src;'
)
);
);
);
Procedure to Run
tbuild -C -f kafka_multi.tpt -v jobvar_multiinstance -j kafka15AccessModuleKafka returns the following error and terminates if the user provides fewer partitions than there are instances:
Dataconnector Instances: %d are greater than total partition count: %d
AccessModuleKafka splits the total 16 partitions and distributes 2 partitions to 8 TPT FileWriter instances equally.
KAFKA_AXSMOD: Running in Multi-Instance mode. This is Instance 8 of 8 KAFKA_AXSMOD: Partition : 1 and Instance number : 8 KAFKA_AXSMOD: Partition : 0 and Instance number : 8 KAFKA_AXSMOD: Running in Multi-Instance mode. This is Instance 3 of 8 KAFKA_AXSMOD: Partition : 11 and Instance number : 3 KAFKA_AXSMOD: Partition : 10 and Instance number : 3 KAFKA_AXSMOD: Running in Multi-Instance mode. This is Instance 6 of 8 KAFKA_AXSMOD: Partition : 5 and Instance number : 6 KAFKA_AXSMOD: Partition : 4 and Instance number : 6 KAFKA_AXSMOD: Running in Multi-Instance mode. This is Instance 2 of 8 KAFKA_AXSMOD: Partition : 13 and Instance number : 2 KAFKA_AXSMOD: Partition : 12 and Instance number : 2 KAFKA_AXSMOD: Running in Multi-Instance mode. This is Instance 4 of 8 KAFKA_AXSMOD: Partition : 9 and Instance number : 4 KAFKA_AXSMOD: Partition : 8 and Instance number : 4 KAFKA_AXSMOD: Running in Multi-Instance mode. This is Instance 5 of 8 KAFKA_AXSMOD: Partition : 7 and Instance number : 5 KAFKA_AXSMOD: Partition : 6 and Instance number : 5 KAFKA_AXSMOD: Running in Multi-Instance mode. This is Instance 7 of 8 KAFKA_AXSMOD: Partition : 3 and Instance number : 7 KAFKA_AXSMOD: Partition : 2 and Instance number : 7 KAFKA_AXSMOD: Running in Multi-Instance mode. This is Instance 1 of 8 KAFKA_AXSMOD: Partition : 15 and Instance number : 1 KAFKA_AXSMOD: Partition : 14 and Instance number : 1