Active Directory Scan: Continuous Loading of Transactional Data - Parallel Transporter

Strategy

Consider the following when setting up a job for Active Directory Scan:

• Specify the attribute names and values for the standard attributes required for the DataConnector operator; FileName, Format, IndicatorMode (where required), and TextDelimiter (required if format is “delimited”).

  For information on use of these standard attributes, see the chapter on the DataConnector operator in Teradata Parallel Transporter Reference (B035-2436).

• Use the wildcard character ( * ) for the FileName attribute according to one of the following strategies:
  • Specify “*” to instruct the DataConnector operator to scan and extract data from all files in the directory.
  • Specify “abc.*” to instruct the DataConnector operator to scan for all files in the directory having file names that begin with the specified character string.
• Specify the directory to be scanned using the DirectoryPath attribute, in the form:

  DirectoryPath=<PathName>

• Use the ArchiveDirectoryPath attribute to specify the path for the archive directory. Once files in the directory have been scanned and their data has been extracted, this specification will cause the files to be moved from the directory identified in the DirectoryPath attribute to that specified in ArchiveDirectoryPath attribute, in order to keep the files from being scanned again.
• Use the DataConnector Vigil attributes to set up the time constraints for the directory scan, as follows:

  Attribute	Setup Requirements
  VigilStartTime	Required to specify the start time for the initial directory scan.
  VigilStopTime	Specifies the time after which no more scans will begin. Any scan that begins before the stop time will run to completion. This attribute is interchangeable with the VigilElapsedTime attribute. Using one of these two attributes is required.
  VigilWaitTime	Specifies the time in seconds between the beginning of one scan and the beginning of the next scan.
  VigilElapsedTime	Specifies the total time in minutes the job will scan the directory for new files in intervals defined by VigilWaitTime. Any scan that starts before the end of the specified elapsed time will run to completion.

  For required syntax and detailed descriptions for all DataConnector attributes, see the Teradata Parallel Transporter Reference (B035-2436).

Active Directory Scan Options

The following options are available to further customize an Active Directory Scan.

• Use several DataConnectors operating in parallel to monitor multiple data sources.
• Use multiple instances of Stream operator to INSERT data into a Teradata Database table at an optimal rate.
• Important optional attributes:
  • Specify the VigilSortFile attribute and set it to TIME to sort files according to the time they were last modified.
  • Specify the VigilNoticeFileName attribute with a file name, so that when the scan file is updated with new data, a notification will be placed in that file.
  • Specify VigilMaxFiles to define the maximum number of files that can be scanned in one pass.
• Multiple schemas:

  When the data from the sources are not all described by UNION-compatible schemas, use column selection and/or derived columns in the Select clauses in the APPLY statement to put UNION-compatible data on the output data streams.

For a typical application of Active Directory Scan, see Job Example 9: Active Directory Scan.

For the sample script that corresponds to this job, see the following script in the sample/userguide directory:

PTS00015: Active Directory Scan.