It isn’t too much of a surprise that Blue Waters, NCSA’s 10+ PFLOPS super expected to come online in 2011, will run Linux. According to the November 2009 list, 89% of the Top500 (or 446 systems) use some variant of that ubiquitous OS. But it is perhaps a little more surprising that the system, a POWER7-based IBM that grew out of DARPA’s HPCS program, will not run AIX.
Staff at NCSA and IBM evaluated both the open-source Linux and IBM’s AIX as potential operating systems for Blue Waters, extensively testing both on identical hardware at large scale. SARA Computing and Networking Services in the Netherlands and the U.S. National Center for Atmospheric Research (NCAR) shared their experiences and provided access to their large IBM POWER6 systems, both with more than 3,300 cores and both using Infiniband interconnects. Both ran IBM’s full HPC Software stack and used the GPFS file system. SARA’s system uses Linux, while NCAR uses AIX.
The team, led by HPC rock star Bill Kramer, looked at “more than 160 features” of both Linux and AIX that are expected over the five year operational life of the system before making a decision, and ran a bunch of application performance tests as well.
NCSA also conducted multiple application performance tests, including MILC, PARATEC, four of the NAS Parallel Benchmarks (Fast Fourier Transform, Conjugate Gradient, Lower-Upper symmetric Gauss-Seidel, and MultiGrid) for Class B and Class D problem sizes, HPC Challenge Benchmarks, and parallel 3D Fast Fourier Transform tests.
…”We determined that either OS could ably meet the requirements for Blue Waters and serve the science and engineering research community,” Kramer said. “Linux offered some additional advantages, such as being somewhat more familiar to the research community and offering the potential for NCSA to more directly participate in enhancing the OS.”
But which distribution, you ask? Well, I asked too, and was told that that discussion is still underway.