Not all problems with data integrity are the result of input errors or semantic integrity violations. Problems originating in disk drive and disk array firmware can also corrupt user data, typically at the block or sector levels. Block- and sector-level errors are the most common origins of disk I/O corruption in user data.
Disk I/O data corruption is typically not detected immediately. Therefore, queries against the corrupted data return answer sets that are semantically correct but factually incorrect, and update or delete operations can miss relevant rows or can change rows in error. When the system detects the corruption, the affected AMP is typically taken offline, utilities such as ScanDisk and CheckTable are performed, and the data is repaired or reloaded. Each of these actions removes access to, or reduces the availability of, the data warehouse to its users until corrections have been made.
Levels of Disk I/O Integrity Checking
To protect against physical data corruption, Vantage permits you to select levels of disk I/O integrity checking of your table and join index data. Secondary index subtables assume the level of disk I/O integrity checking that is defined for their parent table or join index. These checks detect corruption of disk blocks using one of the following integrity methods, which are ranked in order of their ability to detect errors.
- Full end-to-end checksums.
Detects lost writes and most bit, byte, and byte string errors.
- Statistically sampled partial end-to-end checksums.
Detects lost writes and intermediate levels of bit, byte, and byte string errors.
- No checksum integrity checking.
Detects forms of lost writes using standard file system metadata verification.
Disk I/O Integrity Checking Only Detects and Logs Errors
This feature detects and logs disk I/O errors but does not fix them. Upon detecting data corruption, the system removes the affected AMP from service and you must then repair the corrupted data.