KAFKA_AXSMOD: Message Consumption Information Starts ------> KAFKA_AXSMOD: Partition : 2, Start Offset: 0, End Offset : 3479, No. of message(s) read : 3480 KAFKA_AXSMOD: Partition : 1, Start Offset: 0, End Offset : 3400, No. of message(s) read : 3401 KAFKA_AXSMOD: Partition : 0, Start Offset: 0, End Offset : 3358, No. of message(s) read : 3359 KAFKA_AXSMOD: Message Consumption Information Ends <------
This is a count of the messages read and and delivered to the TPT DC by the Kafka Access Module per partition.
If the checkpointing feature (-z option) is used, both CheckPoint Information and Message Consumption Information will be logged.
See the following example:
KAFKA_AXSMOD: CheckPoint Information Starts -----------> KAFKA_AXSMOD: Partition : 2, Start Offset: 0, Message Offset stored at last checkpoint in TPT: 2178, No. of message(s) sent to TPT : 2178 KAFKA_AXSMOD: Partition : 1, Start Offset: 0, Message Offset stored at last checkpoint in TPT: 2178, No. of message(s) sent to TPT : 2178 KAFKA_AXSMOD: Partition : 0, Start Offset: 0, Message Offset stored at last checkpoint in TPT: 1505, No. of message(s) sent to TPT : 1505 KAFKA_AXSMOD: CheckPoint Information Ends <----------- KAFKA_AXSMOD: Message Consumption Information Starts ------> KAFKA_AXSMOD: Partition : 2, Start Offset: 0, End Offset : 2177, No. of message(s) read : 2178 KAFKA_AXSMOD: Partition : 1, Start Offset: 0, End Offset : 2177, No. of message(s) read : 2178 KAFKA_AXSMOD: Partition : 0, Start Offset: 0, End Offset : 2177, No. of message(s) read : 2178 KAFKA_AXSMOD: Message Consumption Information Ends <------
The use case for Checkpoint Information is that it allows the user to ensure/validate that the subsequent restart job begins at the last checkpoint taken in the unsuccessful job.
The following example illustrates statistics of a TPT job that had to be run twice in order to successfully consume all of the messages available in the Kafka Topic.
Run1 (Failed due an error, statistics snipped from its log has been mentioned here):
KAFKA_AXSMOD: CheckPoint Information Starts -----------> KAFKA_AXSMOD: Partition : 2, Start Offset: 0, Message Offset stored at last checkpoint in TPT: 2178, No. of message(s) sent to TPT : 2178 KAFKA_AXSMOD: Partition : 1, Start Offset: 0, Message Offset stored at last checkpoint in TPT: 2178, No. of message(s) sent to TPT : 2178 KAFKA_AXSMOD: Partition : 0, Start Offset: 0, Message Offset stored at last checkpoint in TPT: 1505, No. of message(s) sent to TPT : 1505 KAFKA_AXSMOD: CheckPoint Information Ends <----------- KAFKA_AXSMOD: Message Consumption Information Starts ------> KAFKA_AXSMOD: Partition : 2, Start Offset: 0, End Offset : 2177, No. of message(s) read : 2178 KAFKA_AXSMOD: Partition : 1, Start Offset: 0, End Offset : 2177, No. of message(s) read : 2178 KAFKA_AXSMOD: Partition : 0, Start Offset: 0, End Offset : 2177, No. of message(s) read : 2178 KAFKA_AXSMOD: Message Consumption Information Ends <------
Message Consumption Information displays the total number of messages read and delivered to TPT DC by the Kafka Access Module.
The user should disregard Message Consumption statistics for a failed job because the job failed before all messages read and delivered by Kafka Access Module could be loaded into Analytics Database by the TPT load operators.
The CheckPoint Information is important for failed TPT jobs that use the checkpointing functionality. The CheckPoint Information shows the offsets from where the subsequent restart job will begin. The last CheckPoint Information is available in the 3rd column named "Message Offset stored at last checkpoint in TPT" in the "CheckPoint Information" statistics log.
The "No. of message(s) sent to TPT" till the last checkpoint were successfully loaded.
It can be inferred from the "CheckPoint Information" statistics log that the first run of TPT job loaded (2178+2178+1505= 5861) messages successfully.
Run 2 (Restart job and has been successful, statistics snipped from its log has been mentioned here):
KAFKA_AXSMOD: CheckPoint Information Starts -----------> KAFKA_AXSMOD: Partition : 2, Start Offset: 2178, Message Offset stored at last checkpoint in TPT: 3480, No. of message(s) sent to TPT : 1302 KAFKA_AXSMOD: Partition : 1, Start Offset: 2178, Message Offset stored at last checkpoint in TPT: 3401, No. of message(s) sent to TPT : 1223 KAFKA_AXSMOD: Partition : 0, Start Offset: 1505, Message Offset stored at last checkpoint in TPT: 2867, No. of message(s) sent to TPT : 1362 KAFKA_AXSMOD: CheckPoint Information Ends <----------- KAFKA_AXSMOD: Message Consumption Information Starts ------> KAFKA_AXSMOD: Partition : 2, Start Offset: 2178, End Offset : 3479, No. of message(s) read : 1302 KAFKA_AXSMOD: Partition : 1, Start Offset: 2178, End Offset : 3400, No. of message(s) read : 1223 KAFKA_AXSMOD: Partition : 0, Start Offset: 1505, End Offset : 3358, No. of message(s) read : 1854 KAFKA_AXSMOD: Message Consumption Information Ends <------
Message consumption information is important in the second run as the Job was successful and all the messages read and delivered by Kafka Access Module were loaded into Analytics Database by the TPT load operators.
The user should disregard CheckPoint Information section as the subsequent job will not restart.
It can be inferred from the "Message consumption" statistics log that the second run of TPT job loaded (1302+1223+1854= 4379) messages successfully.
Number of messages successfully loaded in the 1st Run : 5861
Number of messages successfully loaded in the 2nd Run : 4379
Total Number of messages successfully loaded in both the runs : ( 5861 + 4379 ) = 10240
If the TPT job successfully consumes all of the messages in the Kafka Topic in 'n' runs, then the total number of messages read and loaded in those 'n' runs will be equal to the total number of messages available in the Kafka Topic.
The following statistics will be logged when Kafka Access Module is used as a Producer:
KAFKA_AXSMOD: Message Production Information Starts ------> KAFKA_AXSMOD: Partition : 2, No. of message(s) written : 214 KAFKA_AXSMOD: Partition : 1, No. of message(s) written : 213 KAFKA_AXSMOD: Partition : 0, No. of message(s) written : 213 KAFKA_AXSMOD: Message Production Information Ends <------
The information does not include start and end offsets as many producers can write on the same Topic simultaneously. When many producers write to the same Kafka Topic, the messages from each producer are not written in order, and the total number of messages written does not equal the end offset minus the start offset.