Starting with TTU 16.20, Teradata Access Module for Azure Blob supports multiple DataConnector instances.
You can export data parallelly using multiple instances of TPT DC operator from Teradata to blob storage with higher throughput and import data using multiple instances of TPT DC operator.
Teradata Access Module for Azure Blob receives information about the number of TPT DC instances and instance number of each TPT DC instance number from TPT.
The access module appends the instance numbers to the blob object files.
- In the export job, it appends the instance number to the blob object files which allows writing data to multiple blob object files parallelly.
- In the import job, it appends the instance number to the blob object files so that every tpt dc instance will have a unique blob object to read.
Example
DEFINE JOB EXPORT_FROM_TERADATA
DESCRIPTION 'Export data from Teradata to Azure blob storage using Multiple FileWriter instance'
(
STEP EXPORT_THE_DATA
(
APPLY TO OPERATOR ( $FILE_WRITER()[8]
ATTR
(
AccessModuleName = 'libazureaxsmod.so',
AccessModuleInitStr = '-Container <container> -Prefix <prefix> -Object <blobojectfile> -SinglePartFile True'
)
)
SELECT * FROM OPERATOR ( $EXPORT
ATTR
(
SelectStmt = 'SELECT * FROM S3W001_src;'
)
);
);
);
Procedure to Run
tbuild -C -f <.tpt script> -v <Job config file> -j <Log file name>Output
Teradata Access Module for Azure Blob will create 8 blob object files and the export job will be performed parallelly using 8 TPT DC instances.
[root@sdl58678 AWS_TPG]# tbuild -C -f TPT_EXPORT_WITHMultiInstance.tpt -v jobvar_multiinstance -j explog22 Teradata Parallel Transporter Version 20.00.00.00 64-Bit The global configuration file '/opt/teradata/client/20.00/tbuild/twbcfg.ini' is used. Log Directory: /opt/teradata/client/20.00/tbuild/logs Checkpoint Directory: /opt/teradata/client/20.00/tbuild/checkpoint Job log: /opt/teradata/client/20.00/tbuild/logs/explog22-100.out Job id is explog22-100, running on sdl58678 Teradata Parallel Transporter DataConnector Operator Version 20.00.00.00 $FILE_WRITER[7]: Instance 7 directing private log report to 'FILE_WRITER_LOG-7'. $FILE_WRITER[4]: Instance 4 directing private log report to 'FILE_WRITER_LOG-4'. $FILE_WRITER[6]: Instance 6 directing private log report to 'FILE_WRITER_LOG-6'. $FILE_WRITER[5]: Instance 5 directing private log report to 'FILE_WRITER_LOG-5'. $FILE_WRITER[3]: Instance 3 directing private log report to 'FILE_WRITER_LOG-3'. AZUREAXM [613786]: 'MODULE_NAME=Teradata Access Module for Microsoft Azure' Version: MODULE_VERS=20.00.00.00. AZUREAXM [613786]: Copyright 2023 Teradata. All rights reserved. AZUREAXM [613786]: Compiled for PLATFORM_ID=Linux x8664 (64 bit) on Jun 12 2023 .... AZUREAXM [613783]: AXMS3 Library built on Feb 8 2023 02:20:13 AZUREAXM [613783]: Initialization String is '-Container telsabucket01 -Prefix azuretest -Object multi_part_exp-8 -SinglePartFile True' .... $EXPORT: connecting sessions $EXPORT: sending SELECT request $EXPORT: entering End Export Phase $EXPORT: Total Rows Exported: 5237120 $EXPORT: Total Rows Discarded: 0 AZUREAXM [613786]: EOF Statistics: 51.814 megabytes sent to Azure Blob Storage in 10.6 secs, 4.87 megabytes/sec, 0.8 sec user 0.1 sec system AZUREAXM [613785]: EOF Statistics: 51.814 megabytes sent to Azure Blob Storage in 10.7 secs, 4.86 megabytes/sec, 0.8 sec user 0.1 sec system AZUREAXM [613783]: EOF Statistics: 51.815 megabytes sent to Azure Blob Storage in 10.7 secs, 4.84 megabytes/sec, 0.8 sec user 0.1 sec system AZUREAXM [613784]: EOF Statistics: 51.814 megabytes sent to Azure Blob Storage in 10.7 secs, 4.85 megabytes/sec, 0.8 sec user 0.1 sec system .... $FILE_WRITER[8]: Number of records processed: 652669 $FILE_WRITER[5]: Number of records processed: 652671 $FILE_WRITER[7]: Number of records processed: 652670 $FILE_WRITER[1]: Total files processed: 8. $EXPORT: sessions disconnected $EXPORT: Performance metrics: $EXPORT: MB/sec in Export phase: 41.579 $EXPORT: Elapsed time from start to Select phase: 1 second(s) $EXPORT: Elapsed time in Select phase: 3 second(s) $EXPORT: Elapsed time in Export phase: 8 second(s) $EXPORT: Elapsed time from Export phase to end: 4 second(s) $EXPORT: Total processor time used = '0.714933 Second(s)' $EXPORT: Start : Tue Aug 8 01:42:00 2023 $EXPORT: End : Tue Aug 8 01:42:16 2023 Job step EXPORT_THE_DATA completed successfully Job explog22 completed successfully Job start: Tue Aug 8 01:42:00 2023 Job end: Tue Aug 8 01:42:16 2023