16.10 - Initialization String - Access Module

Teradata Tools and Utilities Access Module Reference

prodname
Access Module
vrm_release
16.10
created_date
July 2017
category
Programming Reference
featnum
B035-2425-077K

The following are the parameters of the Teradata Access Module for Amazon S3 initialization string for loading and exporting data using the Teradata Access Module for Amazon S3.

The initialization string consists of a series of keyword and value pairs separated by blanks. The value of a keyword can be an integer or string.

|--------------------------------------------------------------|
| |
|---------S3ConfigDir=ConfigDirectoryPath----------------------|
| |
|---------S3Profile=ProfileName--------------------------------|
| |
|---------S3Region=RegionName----------------------------------|
| |
|---------S3Bucket=BucketName----------------------------------|
| |
|---------S3Prefix=PrefixName----------------------------------|
| |
|---------S3Object=ObjectName----------------------------------|
| |
|---------S3ConnectionCount=n (an integer value)---------------|
| |
|---------S3BufferCount=n (an integer value)-------------------|
| |
|---------S3BufferSize=n (an integer value)--------------------|
| |
|---------S3SinglePartFile=True/False--------------------------|
| |
|---------S3AccessID=AccessKeyId-------------------------------|
| |
|---------S3AccessKey=SecretAccessKey--------------------------|
|--------------------------------------------------------------|

where the following is true:

Syntax Element Description
S3ConfigDir This is an optional parameter. AWS provides for application programs to obtain some frequently needed parameters from two files, config and credentials, that are stored in a configuration directory. By convention this directory is $HOME/.aws, but may be overridden with this parameter.
S3Profile The config and credentials files are broken into sections per AWS documentation. Each section is labeled with a bracketed heading. The S3Profile variable selects a section from each of these two files. There is always a section called [default]. This section will be used if no S3Profile is specified or if the value default is explicitly listed as default: S3Profile=default.
S3Region This is a required parameter. This parameter specifies the Amazon S3 region where the S3 objects are stored. If the Amazon S3 region name is us-west-2, then you need to specify S3Region=us-west-2 in the initialization string or alternatively you can specify region = us-west-2 in the config file. This required parameter will be obtained from the config file unless overridden by being specified explicitly in the access module initialization string.
The original AWS S3 region on the US East Coast is often referred to as “US Standard”. This should be specified as us-east-1.
S3Bucket This is a required parameter. This parameter specifies the Amazon S3 bucket to be used for load and export operations.

This parameter must be listed in the access module parameters initialization string.

S3Prefix This is an optional parameter. This parameter will be prepended to the S3Object string to create the pathname within the bucket. As a convenience in writing export or load scripts, this parameter could be used for the portion of the pathname that does not change when a series of tables are exported or loaded.
Within an S3 bucket there is no need to explicitly create directories. The S3 CLI and GUI will segregate the listing of objects based on the presence of “/”, but this is a display convenience only. You do not need to create directories to use a “/” in a pathname, and the presence or location of one or more “/” in your pathname can come from either or both the S3Region or S3Object strings. In particular, the S3Prefix, if used, does not need to contain or end in a “ / ”.
S3Object This is a required parameter. The S3Prefix, if specified, is prepended to this parameter to generate the pathname of the object to be created.

In order to support large result sets, the preferred mode of operation is to create a series of objects based on the pathname specified by appending a suffix generated by connector. For details, see the description of S3SinglePartFile, below.

S3ConnectionCount This optional parameter specifies the number of TCP connections to the Amazon S3 service.

If not specified, the default of 10 is used.

S3BufferCount This optional parameter specifies the number of buffers to be used with the TCP connections specified by the S3ConnectionCount.

Twice the number of connections is the minimum recommended value. If not specified, the value of (2*S3ConnectionCount) is used as default.

S3BufferSize This optional parameter specifies the size of the buffers to be used for the TCP connections. The default is 8MB (8388608 bytes). For export operations the buffersize must be at least 5 megabytes (5MB). For convenience the following multipliers can be used: k (1000), K (1024), m (1000 * 1000), and M (1024*1024). For example, the default could be specified as 8M or 8192K.
S3SinglePartFile This optional parameter determines whether the S3 object being read or written is a single file with the name specified by S3Object or a set of numbered files in an “apparent” directory named by the S3Object. As discussed above, directories don’t really exist, but the GUI and CLI segregate files based on “/” making it appear that there is a directory.

When an application creates an object on S3, the object can be created from up to 10,000 “pieces”. The size and SHA hash of the “pieces” must be presented when the connection is opened. Since it is not desirable to buffer the file on disk, the size of a “piece” is limited to the size of the buffer. The minimum piece size/buffer size is 5MB. By default 8MB is used as a good trade-off between memory usage and performance. When the pieces are to be turned into an S3 object (a file) the command is limited to 10,000 pieces. That effectively limits the size of a single S3 object (file) that is created to 80 Gigabytes.

The solution is to provide a mechanism to automatically treat a list of objects (files) with related pathnames as a single file for the export or load operation. The connector appends a “/”, followed by an “F”, followed by a sequence number. So there is an “apparent” directory created with the name of the requested S3 object. The files in that “apparent” directory are named F000001, F000002, F000003, etc. This solves the problem of being able to handle an arbitrarily large result set from the database. The suffixes of the “parts” are predefined, and cannot be overridden.

If it is possible that the output will exceed 80 Gigabytes, the S3SinglePartFile parameter must be set to false so a set of numbered files can be created. The default value for S3SinglePartFile is "false".

The S3SinglePartFile parameter may be set to "true" in the following scenarios:
  • For tests using small datasets
  • For production where the customer is absolutely certain the result set to be stored is smaller that 10,000*buffersize

The S3SinglePartFile must be set to true when reading a single S3 object (file) of arbitrary size that isn’t split up into parts (F000001, F000002, etc.).

S3AccessID This is a required parameter. This is effectively the Amazon S3 user id. The value of S3AccessID can be specified via the initialization string parameter S3AccessID, environment variable AWS_ACCESS_KEY_ID, or AWS credentials file. If specified by initialization string parameter S3AccessID, the value of environment variable AWS_ACCESS_KEY_ID and contents of AWS credentials file will not be considered. If set by Environment Variable AWS_ACCESS_KEY_ID, the contents of AWS credentials file will not be considered.

Please consider your security policy when deciding if you want to store this value in your JV file.

S3AccessKey This is a required parameter. This is effectively the Amazon S3 password. Value of S3AccessKey can be specified via initialization string parameter S3AccessKey, Environment Variable AWS_SECRET_ACCESS_KEY, or AWS credentials file. If specified by initialization string parameter S3AccessKey, the value of environment variable AWS_SECRET_ACCESS_KEY and contents of AWS credentials file will not be considered. If specified by Environment Variable AWS_ACCESS_KEY_ID, the contents of AWS credentials file will not be considered.

Please consider your security policy when deciding if you want to store this value in your JV file.