Batch Directory Scan
Batch Directory Scan uses multiple DataConnector operator instances to scan an external directory of flat files, searching for files that match the wildcard specification in the FileName attribute.
When the scan is complete, DataConnector places the data in the data stream for use by the consumer operator in the next job step. No further scanning is done, and any data added to the flat files after the scan will not be picked up until the next time the job is run.
Use the following strategy when setting up the Batch directory scan:
For the sample script that corresponds to this job, see the following script in the sample/userguide directory:
PTS00014: Batch Directory Scan.
Note: The Batch Directory Scan functionality is supported when using the HDFS API interface to process Hadoop files, but is not supported when using the TDCH-TPT interface to process Hadoop files and tables. For more information, see “Processing Hadoop Files and Tables” in Chapter 3 of the Teradata Parallel Transporter Reference.