NetworkWorld posted a feature article this morning on Bruce Allen, astrophysicist turned supercomputer manufacturer. Since 1998, Allen has hand-built three clusters in order to further his research in observing theoretical gravitational waves. Allen’s latest contribution to hand-built machines has landed at #79 on the current Top500 list. The 6,000+ core machine is held together by a gigabit Ethernet backbone, all hand-laid by Allen and his staff at the Max Planck Institute. So why all the work?
If you go to a company — Dell or IBM — and you say, ‘I’ve got a $2 million budget, what can you sell me for that price?’ you’ll come back with a certain number of CPUs,” he says.
“If you then go and look at Pricewatch or some other place where you can find out how much the gear really costs, you find out that if you build something yourself with the same money you’ll end up with two or three times the processing power.”
Allen’s success with this method goes back to 1998 when he used an NSF grant to purchase some Sun workstations. Rather than buying the workstations, he bought some DEC Alpha machines cheap because they were near end-of-life.
I’ve personally seen and participated in many such grassroots cluster projects. I remember surfing the Fry’s ads looking for specific motherboards on sale. However, it takes a special amount of patience to take this road. Building and supporting machines consisting of 6,000+ cores is very labor intensive. I’ve often thought about the countless hours I spent in the lab as a fuzzy-haired college student debugging `bootp` issues on 10base Ethernet. Is it worth the effort to do so, or is the vendor-provided turnkey solution really pay off at the end of the day?
I’ll let the audience answer that one. In the mean time, you can read the full article here.
Academics don’t have to account for labor costs when they assess the total cost of their clusters. If that is your situation, then it makes a lot of sense to shift costs from vendors, who charge actual money and therefore show up in the project budget, to faculty and graduate students, who don’t. If, on the other hand, you have to charge labor costs against the cluster project, then you see that the time you spend chasing down deals on pricewatch (and dealing with hardware that isn’t *quite* identical between nodes) quickly eats up any savings you might have hoped to gain.
Which method is better? In effect Allen is hiring astrophysicists to do system integration work, which is likely to be less efficient than farming that work out to people who specialize in it. Moreover, much of the hidden cost is borne by his graduate students in the form of longer working hours and longer time to graduation, which seems a bit exploitative to me (full disclosure: I’m a PhD in astrophysics who spent his share of time in graduate school scrounging up computing resources). For those reasons, I lean toward fully accounting for all of the costs involved in building a system, including those that are otherwise hidden in your routine payroll expenses.
no, it’s not much about labor costs, though the pop media loves to fixate on that (will pay pizza to build my supercomputer!) the real issues are more subtle – things like how Big Name Vendors (BVNs) only want to build clusters with certain designs or components, or how a BNV’s product lines might be tuned for a different kind of computing than you have. for instance, big Gb clusters are fairly unusual – people who want big parallel clusters tend to also want fast interconnect.
sure, you don’t want your physicists spending weeks unboxing motherboards, so you get someone to do it for you. but a local low-overhead whitebox vendor will do as good a job as a BNV. sure, you want to be careful choosing your parts, and doing so requires some attention. with a BNV, you don’t get to choose, but can expect them to resolve issues that might arise through a support contract. but BNVs can’t guarantee that everything will work perfectly either, not much more than a no-name. heterogeneity is not something inherent to DIY or absent from the BNV approach.
besides more flexibility in config/design, BNVs also tend to lag in time – you’ll usually be able to get a processor, chipset. interconnect or disk earlier on the open market.
if you really want a turnkey cluster, BNV may be a very good idea, since it means you don’t deal with matching your mpi to your driver to your nic to your switch, etc. or even Small Name Vendors, which tend to specialize in HPC more. but since the whole stack is open and/or commoditized now, you _can_ DIY and with some care, not spend much time or effort getting it right. in that sense, the article’s mention of gigabit is right on target: an interconnect that’s by far the most standardized and widely understood, with low support costs, not just upfront prices…
Hi rpl,
It’s easy for me to account for the ‘hidden labor costs’ since they are not at all hidden! I see every cent of them on my monthly and annual budgets.
I have two full-time professional system administrators (Carsten Aulbert and Henning Fehrmann). Both have PhDs in physics, and both spent a number of years as graduate students using clusters and helping with their care and feeding. Carsten and Henning are assisted by a handful of ruthlessly exploited undergraduate assistants. Total labor costs per unit time are between ten and fifteen percent of the amount that the cluster depreciates in the same period! They are less than ten percent of the cluster depreciation cost plus the electrical costs.
Note that even a cluster from a BNV would require at least one full-time sysadmin. And you’d need a backup if that person ever wanted to take a vacation, have some family time, etc.
Cheers,
Bruce Allen