Today Chris Aycock (figuratively) dropped by with an essay on forces shaping the HPC market’s product offerings. The bottom line? “Crap+1″ almost always sells better than “shiny new idea 1.0.” — Ed.
Hollywood writers and producers often talk about “high concept,” which is a story idea that can be summarized in a few words and will be understood by a large audience. The tactic usually involves playing a twist on an existing popular film. For example, Alien can be pitched as Jaws in outer-space. These films are often derided for being more about star power and special effects than for having deep characters and a rich plot. These films also, however, make money at the box office.
So too is the way of technology in the enterprise. The most successful products tend to resemble “crap + 1″ because they are more easily adopted. The marketing for the original Opteron may well have been: “it’s x86, but 64 bit!” Contrast that with Itanium’s approach of attempting something more compelling, which so far has had a dismal sales record. (That’s not to say that Intel was totally shutout; they quickly realized that customers wanted 64-bit x86 and were soon able to take back market share from AMD.)
Likewise, many successful software projects are based on previous winners. GNU/Linux started as an open-source clone of Unix; BeOS, in comparison, tried to be new and different.
The reason film studios love high-concept scripts is that potential audiences are more likely to buy tickets when they feel they’ll understand and enjoy the movie. Similarly, successful technology companies know that customers are more likely to take a chance on a new product when they believe that the time, costs, and effort required for adoption are low. This implies that the existing data center must be respected.
One of the ironies of entrepreneurship is that a start-up’s most reliable revenue stream usually comes from big companies. The simple reason is that Fortune 500-sized corporations have the resources needed to test-out new and expensive technology. This of course leads to a paradox because the large firms tend to have a lot of legacy infrastructure. Thus the innovative start-up must actually build on existing platforms rather than design something wholly new.
As an example, PathScale sold network interface cards that plugged into existing InfiniBand fabric. This market entrance was compelling enough for QLogic to buy out the hardware. In contrast, whole-machine companies like SiCortex–who bought out PathScale’s software–will have a harder time finding wide-spread adoption beyond universities and government labs. (I will say right now that I believe this is a shame since I think SiCortex has one of the most compelling products out there. But alas, a mere observer such as myself can no more change the free market than a physicist can bend the laws of gravity.)
High concept does point to possible areas of innovation though. Given the buzz of high-productive computing, companies that innovate on existing domain-specific languages will probably be more successful. Kx System’s q/kdb+ is an array language made to look like SQL. Interactive Supercomputing’s Star-P parallelizes MATLAB and Python. For that matter, Star-P is now available as an on-demand product, which makes HPC adoption even easier. (Even Google is getting into on-demand high-performance Python with their App Engine.)
Any entrepreneur looking into HPC should make sure that his product lowers the barriers for customer adoption. Building on the existing data center is a smart way to a Hollywood ending.