Joe Landman from Scalable Informatics writes about why the delivered performance of disk subsystems doesn’t look anything like marketing specifications.
The overhead is the killer. Most people building systems really don’t understand how much of a killer it is. IO is usually relegated to secondary or tertiary consideration, though as data sets grow large, this is a very bad/dangerous thing to do. Moreover, expectations of performance are far … far out of whack with reality. What a user seeing 100MB/s on a single threaded streaming case might expect for a random case, shouldn’t be 100MB/s as well (for spinning rust disks). For some of the simpler tests we are doing (random 32k reads and writes to mimic a particular use case), we are seeing from 1-10 MB/s on real hardware.
Read the Full Story.