Careful with that measurement

Patrick Madden has a great post on his Amdahl’s Law blog about the be careful when measuring performance (or believing someone else’s measurement).

He uses the example of matrix multiplication on the Cell processor, to show linear performance scaling to 4 SPE’s with a moderately big matrix. Patrick then goes on to show us all the trap we have fallen into – that the kernel solution time as reported by the software scales linearly – but that the total solution time (i.e. from starting the program to it finishing) as reported by Unix ‘time’ hardly scales at all.

The point is that we need to be careful what we measure – and not forget the time required for the often large parts of the code (and supporting OS activity such as creating threads etc) that get us to the computational kernel in the first place.