COTS supercomputing: point/counterpoint

Print Friendly, PDF & Email

Last week HPCwire ran an article by Michael Wolfe at The Portland Group called Compilers and More: The Dangers of COTS Supercomputing. In it the author argues for the point of view that the current HPC market is mired by its wedding to commodity technology and lack of investment in both hardware and software.

The HPC ecosystem is in perfect balance, with little investment and innovation in both hardware and software. We’re in a precarious position now. …In response to Rob Pennington, I believe that the HPC market is too small to support an aggressive hardware business, and it’s equally true that it’s too small to support a software tools industry.

Michael doesn’t offer much hope for a resolution, however

…the tools will appear only if and when they apply to a larger market, or if some company (unlikely) or government agency (perhaps likely) chooses to make a long-term strategic investment.

As a person with libertarian leanings, I rarely think big government investments are healthy in the long term.

To round out the conversation, Joe Landman over at has a different view. Joe argues that we have the market we have because that’s the market we were willing to pay for (he’s right). He also argues that this market will continue to respond to the stimuli we give it, but that there is reason to think that we will continue to adapt our techniques to new technologies as they become available (he discusses the specific examples of GPGPUs).

Joe’s argument has the edge of practicality — not so much pathetically waiting for the scraps as they fall off the master’s table, but recognizing that the scraps are going to fall, and we can use that information as part of a larger strategy.

Read both, and decide for yourself. Then leave a comment about what you see ahead for HPC: it is a dead market walking, or just waiting to transform yet again?


  1. I largely agree with Joe – I would describe the market as one which is constantly evolving based on the economics of performance. I think that careful consideration of such a definition answers the question posed as to whether it’s a dead market or just waiting to transform again. It is an evolving market, and always has been.. that is the constant characteristic of it.

    To me, the concern from some of veterns of HPC over the market not catering to their view of how things should be is somewhat similar in nature to that of a parent complaining about ‘kids these days’ — yes, the world is a different place, but is it really worse off? Sure, government funding of hardware and software development is less focused on HPC than it used to be, and thus progress doesn’t seem quite as rapid and far reaching. Maybe this is because much of the low-hanging fruit has already been plucked? Or because, for many people, the current computational hardware and software ecosystems are far enough ahead of the algorithmic and scientific ecosystem that there’s less need, for most people, for the same kind of focused investment of yester-year? These can be debated, of course, but in my opinion, the HPC market isn’t doing half bad.

    I’ll happily expand upon this with examples and analogies if needed, but given my tendency to ramble, I’ll not go there unless dragged into it. And, of course, all due respect given to the veterans of the field – a difference of opinion is a useful thing to have every now and then.

    (Ok, I couldn’t end there, despite my best intentions… I don’t actually view the developments in hardware or software that trickle down from the larger industries into HPC as scraps at all. We can certainly always wish for something faster or better, but looking at Nehalem, for example, I’d argue that the master at the table is feeding us filet mignon, at least from what I’ve seen so far. Could a better chip be designed for HPC workloads? Sure, hence things like QCDOC or DE Shaw’s “Anton” system, but both are highly specialized. For a field that encompasses so much variability, I’d bet it’s virtually impossible to design, manufacture and support hardware that is even remotely close to providing equal performance for the same price. In short, things are good enough as is that such investments are better put into other things. To quote the Stones, “You can’t always get what you want… but if you try some time well you just might find… you get what you need!”)

  2. Brian my man, you win the prize for the Best insideHPC Comment Ever. You can pick up your Chrysler Cordoba at John Leidel’s house the next time you’re in Texas.

    Seriously, thanks.

  3. Hah! Did I win because I quoted the Stones, or because I lack the ability to say anything in a succinct fashion thereby leaving long, rambling comments like the above? 🙂