Why Compilers Love Messing with Benchmarks

Print Friendly, PDF & Email

brendan_rajasthan2011_thumbOver at Brendan Gregg’s Blog, the senior performance architect at Netflix writes that if you want accurate and trustworthy benchmarks, you need to perform active benchmarking, as everything, including compilers, can mess with your benchmark.

If you want to compare different servers using benchmarks that you compile, you need the compilers to match, or you need to take that into consideration. This should be something you unearth by following an active benchmarking approach, where you study and understand what the benchmark really does.

Gregg describes “Passive Benchmarking” as a methodology where Benchmarks are commonly executed and then ignored until they have completed. Contrast this to active benchmarking, where you analyze performance while the benchmark is still running (not just after it’s done), using other tools.

With Active Benchmarking, you can confirm that the benchmark tests what you intend it to, and that you understand what that is. Data becomes Information. This can also identify the true limiters of the system under test, or of the benchmark itself.

Read the Full Story.