Move Over, HPCers: Another Wave of Immigrants Is Hitting Your Shores

Print Friendly, PDF & Email

In this special feature from The Print ‘n Fly Guide to SC13 in Denver, Steve Conway from IDC Research discusses the synergies between Big Data and HPC.

file000005The global HPC market’s enormous expansion from about $2 billion in 1990 to $21.9 billion in 2012 has been heavily driven by successive waves of new users, with each new group of arrivals carrying its own expectations and opportunities. Growth-bringing newcomers have generally been welcomed to the fold by prior HPC users – especially those who remember the post-Cold War market dip of the early 1990s, when there were serious concerns about the long-term viability of supercomputing. Xenophobia from the HPC establishment has been blessedly rare (although I recall giving a talk on SMB users at ISC a few years ago and hearing a few HPC stalwarts in the audience sniff, “That’s not high performance computing.”).

Let’s look at the waves of new users and their requirements. In 1976, the CRAY-1 supercomputer was delivered to its first customer, Los Alamos National Laboratory (LANL), as a blazingly fast hardware platform with no operating system. Not to worry: LANL and others in the first wave of HPC users, primarily government and university researchers, typically had enough in-house technical savvy and personnel to write software themselves when the need was critical.

The second wave of adoption carried HPC into industry, initially the automotive and aerospace sectors, starting in the late 1970’s. These users required HPC vendors to provide not only an operating system and other system software, but to port the key third-party ISV applications needed to run the users’ industry-specific problems — and to run the applications with the reliability expected in production computing environments.

The third important wave of HPC market growth began with the birth of the cluster in the late 1990s and took off in earnest in 2002. Since that year, clusters have almost single-handedly driven the rapid growth in the HPC market and are now the dominant species of technical servers, representing about two-thirds of HPC systems sold each year. The compelling price/(peak) performance of clusters further democratized the HPC market, allowing even SMBs and SMSOs (small and medium-size science organizations) to benefit from this game-changing technology.

And now it’s time to roll out the red carpet for another new wave of users: organizations that are migrating to HPC to perform high performance data analysis (HPDA), sometimes in near-real time, using graph analytics, semantic analysis and other advanced methods.

Examples of large organizations in this category include PayPal, which IDC estimates has saved over $700 million by adopting HPC for real-time detection of online consumer fraud; German pharmaceutical firm Schroedinger, which now screens massive numbers of drug candidates in the public cloud; and the U.S. Government’s Centers for Medicare and Medicaid Services, which has been evaluating how HPC can help to capture more of the $100 million-plus lost annually to fraud in the programs CMS oversees. Smaller first-time adopters come in many shapes and sizes.

Examples here include horizontal players such as Emcien (Atlanta, GA), whose cloud-based fraud detection algorithm is used everywhere from classified government to retailers; and vertical players like Apixio (San Mateo, CA), which applies graphing to heterogeneous medical data to assess risks, help manage patient populations, and balance quality and costs for large providers.

IDC forecasts that thanks to new arrivals like these, along with organic growth, the worldwide HPC market will continue its healthy growth curve to reach $30 billion in 2017. It’s an exciting time to be part of this community.

This story appears in The Print ‘n Fly Guide to SC13 in Denver. We designed this 24-page Guide to be an in-flight magazine custom tailored for your journey to the Mile-High city at SC13. Contents


  1. Sharan Kalwani says

    A very succinct and to the point contemporary description of the HPC market today!