It’s easy for me to account for the ‘hidden labor costs’ since they are not at all hidden! I see every cent of them on my monthly and annual budgets.

I have two full-time professional system administrators (Carsten Aulbert and Henning Fehrmann). Both have PhDs in physics, and both spent a number of years as graduate students using clusters and helping with their care and feeding. Carsten and Henning are assisted by a handful of ruthlessly exploited undergraduate assistants. Total labor costs per unit time are between ten and fifteen percent of the amount that the cluster depreciates in the same period! They are less than ten percent of the cluster depreciation cost plus the electrical costs.

Note that even a cluster from a BNV would require at least one full-time sysadmin. And you’d need a backup if that person ever wanted to take a vacation, have some family time, etc.

Cheers,
Bruce Allen

sure, you don’t want your physicists spending weeks unboxing motherboards, so you get someone to do it for you. but a local low-overhead whitebox vendor will do as good a job as a BNV. sure, you want to be careful choosing your parts, and doing so requires some attention. with a BNV, you don’t get to choose, but can expect them to resolve issues that might arise through a support contract. but BNVs can’t guarantee that everything will work perfectly either, not much more than a no-name. heterogeneity is not something inherent to DIY or absent from the BNV approach.

besides more flexibility in config/design, BNVs also tend to lag in time – you’ll usually be able to get a processor, chipset. interconnect or disk earlier on the open market.

if you really want a turnkey cluster, BNV may be a very good idea, since it means you don’t deal with matching your mpi to your driver to your nic to your switch, etc. or even Small Name Vendors, which tend to specialize in HPC more. but since the whole stack is open and/or commoditized now, you _can_ DIY and with some care, not spend much time or effort getting it right. in that sense, the article’s mention of gigabit is right on target: an interconnect that’s by far the most standardized and widely understood, with low support costs, not just upfront prices…

Which method is better? In effect Allen is hiring astrophysicists to do system integration work, which is likely to be less efficient than farming that work out to people who specialize in it. Moreover, much of the hidden cost is borne by his graduate students in the form of longer working hours and longer time to graduation, which seems a bit exploitative to me (full disclosure: I’m a PhD in astrophysics who spent his share of time in graduate school scrounging up computing resources). For those reasons, I lean toward fully accounting for all of the costs involved in building a system, including those that are otherwise hidden in your routine payroll expenses.