When the system retrieves files (SFTP Inbound), data originates from an external system (in a file format) and gets parsed for injection into Vantage tables.
When an SFTP node is executed, the following scenarios can occur:
- Success scenarioFiles are picked up from the selected SFTP server, matching file name or glob expression defined in the Data integrations module. The glob expression can contain wildcards: * for multiple characters and ? for a single character.When multiple files are processed, they are streamed individually and in alphabetical order.
- Invalid file content
File types are checked that only support processing for delimited files. If the file content is not valid, the workflow fails and logs errors.
- The errors can be viewed on the Workflow status page.
- The log file includes an error message and the skipped file names.
- File processing errorIf, for example, 10 files are picked up and an error occurs on the fifth file, four files are inserted into the database and the remaining six files are skipped. An error message indicates the line number at which the fifth file failed. The Workflow status shows which files are skipped and how many rows are inserted (if at all).
Data is written to selected database table using the following requirements.
- All files read successfully before the failed file are inserted into the database.
- Completely read files will be ignored in subsequent runs (tracked in DI_PROCESSED_FILES table), unless the file is altered (File Edit time stamp and Size is compared).
- Files are sorted by their name before starting to read.
- The workflow logs the error on which file caused the error, and stops processing other files.
- The node error is recorded on the Workflow status page and the log file.
- The file that was skipped due to the error is recorded on the Workflow status page and the log file.
- If the data integration's source SFTP connection is defined with the Include header row option in the file format. The header should be in first row of every input file.
- The first line of the input/source file cannot be empty. Empty lines in the middle and end of the file are ignored.
- Data from the selected source data format field (in file) is written to the mapped destination table field.
- Data fields use the ISO standard when written.
- A new row is created for each line from the parsed file.