The idea of a hosted computing offering (what today is called cloud computing but that’s just a label) can be intellectually appealing, especially to datacenter managers who would like to turn the day-to-day mostly mechanical management of capital expenses on hardware and maintenance of large datacenter infrastructure — things that for most large users of computing are not core concerns — over to someone else and focus on enabling the success of their users based on their organization’s core competencies. But the reality of shifting complex environments with hundreds of applications and thousands of users onto a new infrastructure — any new infrastructure, even one you own — is complex, and letting your data and applications swim in a public pool with everyone else’s data and applications frequently gives people night sweats.
A solution has already been proposed — private clouds — but this one has never made sense to me. You get to replace your infrastructure with another infrastructure you buy, own, and maintain yourself and you have to add and manage a bunch of software in the cloud services stack that you didn’t have to worry about before. This is a good idea how? From a post by Amazon CTO Werner Vogels announcing their new offering
Elasticity is one of the fundamental properties of the cloud that drives many of its benefits. While virtualization has tremendous benefits to the enterprise, certainly as an important tool in server consolidation, it by itself is not sufficient to give the benefits of the cloud. To achieve true cloud-like elasticity in a private cloud, such that you can rapidly scale up and down in your own datacenter, will require you to allocate significant hardware capacity. While to your internal customers it may appear that they have increased efficiency, at the company level you still own all the capital expense of the IT infrastructure. Without the diversity and heterogeneity of the large number of AWS cloud customers to drive a high utilization level, it can never be a cost-effective solution.
Amazon has introduced a kind of middle ground that will allay some of the data and app security concerns while maintaining most of the capex savings, and some of the operational advantages, of using externally hosted computing services. They call it a virtual private cloud, and the idea is that Amazon buys, hosts, and cares for a set of machines that you incorporate directly into your own address space and run as part of your network of resources.
I think this is relevant to HPC readers because it begins to address one of the fundamental concerns that organizations cite when they talk about why they have to own their HPC resources. The big picture here is not one of cloud, or grid, or any of the other fads of the moment. The underlying issue is the development of a viable commercial option for enterprises to adopt HPC technologies that can be effective for their businesses without having to take the “HPC plunge” — that big leap of faith (think of it as an organizational step function that may also come along with a large outlay of cash) a non-HPC organization takes when it decides to disrupt its existing IT environment and established skills set by moving some part of its work onto a cluster of some sort.
Yes, there are probably still all of the performance issues that we’ve already talked about, but as Ian Foster and others have pointed out in comments on this site there are a variety of reasons that people turn to HPC, raw performance on a single large job being only one possible use case. Anyway we’ve already started to see companies like New Servers and Penguin (with their Penguin On Demand facility) offer “bare metal” servers configured with InfiniBand to get around some of the performance disadvantages of more highly evolved environments like Amazon’s EC2.
The development of a viable commercial smooth entry slope into HPC for the vast majority of companies, researchers, and communities not using HPC would be a Really Good Thing. The size of that market could potentially dwarf the size of the current HPC market with is mostly supported by the state (large government investments dominate the HPC business), jump starting our cash-starved industry.
If you’re interested in digging in a little further, GigaOm’s Stacey Higginbotham digs into this announcement as well, and her article is insightful.