Sounds like a bunch of low-end RAID remarketers scrambling around to turn thier product bugs into opportunities for new features and PR.

FYI, for those who haven’t done the math…parity-based raid (single or double parity, RAID 5, 6, etc. — it matters not) with huge capacity disk drives (sata or otherwise) is like playing Russian Roulette with extra bullets in the cylinder.

When a drive fails, you multiply your bit-error rate by the number of disks in the parity group, and there is no telling whether the parity was calculated correctly before being written – so take that number and double it again. Low-end RAID controller manufacturers have known this for many years now. The number one “silent read” error happens when parity is incorrectly calculated by the RAID controller on read or write…and you never know until to need to reconstruct data from that parity (when a disk fails or times-out).

All these RAID manufacturers are touting new “features” to correct their own design failures…and blaming the disk drives in the process. Desktop drives have better BERs and error correction than most other system components…here are the numbers (from Kelemen’s talk):

● NIC/link: 10^-10 (1 bit in ~1.1 GiB)
● DRAM memory: 10^-12 (1 bit in ~116 GiB)
● desktop disk: 10^-14 (1 bit in ~11.3 TiB)
● enterprise disk: 10^-15 (1 bit in ~113 TiB)

Bottom line is that as disks get bigger and bigger, parity-based RAID is a bigger and bigger problem — smart storage architects stick with one of several forms of simple mirroring…especially where desktop class disks are involved. Either way I’ll bet that thorough analysis indicates that RAID parity is the culprit and this all has very little if anything to do with desktop-class vs. enterprise class disks.

We have been dealing with this issue in general at several HPC institutions and this is not a phantom problem that exists. I have published papers from CERN as well as Lawrence Livermore that further define the problems that they face. The intent is to acknowledge that the problem exists. ZFS is a files system which can in fact address this issue on the CPU side however, when fighting corruption in this method there is obviously overhead associated with the checks. I hope this helps and if you would like the documents please let me know.

This is why zfs is interesting, as it may be able to detect/correct some of these errors. But then again, Google’s replication concepts, and other designs, especially the ones that provide fast error detection and correction have to be the “way” of the future. That is, in the limit of large drives and bit error rates small enough that even small arrays will trigger errors, your only real choice is a system that can tolerate errors. In the limit of large numbers of drives, your MTBF for failures approaches 0. You don’t want to be continuously scanning/rescanning for the errors as the scanning may do its own bit flipping, and it doesn’t solve the core problem.