16.10 - Initialization String - Access Module

Teradata Tools and Utilities Access Module Reference

prodname
Access Module
vrm_release
16.10
created_date
July 2017
category
Programming Reference
featnum
B035-2425-077K

The following are the parameters of the Teradata Access Module for Azure initialization string for loading and exporting data using the Teradata Access Module for Azure.

The initialization string consists of a series of keyword and value pairs separated by blanks. Each keyword is preceded by a hyphen, and is not case-sensitive. The value is either an integer or a string.



Two more required parameters StorageAccountName and StorageAccountKey need to be specified in the credentials file under configuration directory, because the information stored in these parameters is sensitive in nature.

Syntax Element Description
-Container (Required) Specifies the Microsoft Azure Blob Storage container to be used for load and export operations. This parameter must be listed in the access module parameters initialization string.
-Prefix (Optional) Is prepended to the -Object string to create the pathname within the container. As a convenience in writing export or load scripts, this parameter could be used for the portion of the pathname that does not change when a series of tables are exported or loaded. The use of this parameter is optional.

Within a Microsoft Azure Blob Storage container there is no need to explicitly create directories. The Microsoft Azure and GUI will segregate the listing of objects based on the presence of “/”, but this is a display convenience only.

You need not create directories to use a “/” in a pathname, and the presence or location of one or more “/” in your pathname can come from -Object strings. In particular, the -Prefix, if used, need not contain or end in a “/”.

-Object (Required) The -Prefix, if specified, is prepended to this parameter to generate the pathname of the object to be created.

In order to support large result sets, the preferred mode of operation is to create a series of objects based on the pathname specified by appending a suffix generated by connector. For details, see the description of -SinglePartFile, below.

-JobProfile (Optional) The credentials file is broken into sections. Each section is labeled with a bracketed heading. The Jobprofile variable selects a section from each of these two files. There is always a section called [default]. This section will be used if no Jobprofile is specified or if the value default is explicitly listed as default: -Jobprofile default.
-SinglePartFile This optional parameter determines whether the Microsoft Azure Blob Storage object being read or written is a single file with the name specified by -Object or a set of numbered files in a “apparent” directory named by the -Object. As discussed above, directories don’t really exist, but the GUI and CLI segregate files based on “/” making it appear that there is a directory.

When an application creates an object on Microsoft Azure Blob Storage, the object can be created from up to 50,000 “pieces”. The size and SHA hash of the “pieces” must be presented when the connection is opened. Since it is not desirable to buffer the file on disk, the size of a “piece” is limited to the size of the buffer. The minimum piece size/buffer size is 4MB. By default 4MB is used as a good trade-off between memory usage and performance. When the pieces are to be turned into an Microsoft Azure Blob Storage object (a file) the command is limited to 50,000 pieces. That effectively limits the size of a single Microsoft Azure Blob Storage object (file) that is created to 200 Gigabytes.

The solution is to provide a mechanism to automatically treat a list of objects (files) with related pathnames as a single file for the export or load opeation. The connector appends a “/” followed by an “F” followed by a sequence number. So there is an “apparent” directory created with the name of the requested Microsoft Azure Blob Storage object. The files in that “apparent” directory are named F000001, F000002, F000003, etc. This completely solves the problem of being able to handle an arbitrarily large result set from the database. The suffixes of the “parts” are predefined and cannot be overridden.

If it is possible that the output will exceed 200 Gigabytes, the -SinglePartFile parameter must be set to false (the default) so a set of numbered files can be created.

The -SinglePartFile parameter may be set to "true" in the following scenarios:
  • For tests using small datasets
  • For production where the customer is absolutely certain the result set to be stored is smaller than 50,000*buffersize.

It must be set to true when reading a single Microsoft Azure Blob Storage object (file) of arbitrary size that isn’t split up into parts (F000001, F000002, etc.).

-ConfigDir (Optional) Specifies the configuration directory which contains the "credential" file. By default this directory is $HOME/azureaxsmod but may be overridden with this parameter.
-ConnectionCount (Optional) Specifies the number of TCP connections to the Microsoft Azure Blob Storage. If not specified, the default of 10 is used.
-BufferCount (Optional) Specifies the number of buffers to be used with the TCP connections specified by -ConnectionCount. Twice the number of connections is the minimum recommended value. If not specified, the value of (2*ConnectionCount) is used as default.
-BufferSize (Optional) Specifies the size of the buffers to be used for the TCP connections. The default is 4MB (4194304 bytes). For export operations the buffersize must be at least 5 megabytes (5MB). For convenience the following multipliers can be used: k (1000), K (1024), m (1000 * 1000), and M (1024*1024). For example, the default could be specified as 4M or 4096K.
-Tracelevel Sets the level of detail to be posted to the log file, where:

0: Disabled – No logging. This is the default if tracelevel is not specified.

1: Events – Events/Requests received by Azure Axsmod from the application.

2: Info – Provides the imported or exported block lengths and performance-related information.

3: ALL – Provides the hexadecimal dump of data which passed through the Teradata Access Module for Azure.

StorageAccountName (Required) This is effectively the Microsoft Azure Blob Storage user id.
StorageAccountKey (Required) This is effectively the Microsoft Azure Blob Storage password.