Amazon introduces virtual private cloud offering

The idea of a hosted computing offering (what today is called cloud computing but that’s just a label) can be intellectually appealing, especially to datacenter managers who would like to turn the day-to-day mostly mechanical management of capital expenses on hardware and maintenance of large datacenter infrastructure — things that for most large users of computing are not core concerns — over to someone else and focus on enabling the success of their users based on their organization’s core competencies. But the reality of shifting complex environments with hundreds of applications and thousands of users onto a new infrastructure — any new infrastructure, even one you own — is complex, and letting your data and applications swim in a public pool with everyone else’s data and applications frequently gives people night sweats.

A solution has already been proposed — private clouds — but this one has never made sense to me. You get to replace your infrastructure with another infrastructure you buy, own, and maintain yourself and you have to add and manage a bunch of software in the cloud services stack that you didn’t have to worry about before. This is a good idea how? From a post by Amazon CTO Werner Vogels announcing their new offering

Elasticity is one of the fundamental properties of the cloud that drives many of its benefits. While virtualization has tremendous benefits to the enterprise, certainly as an important tool in server consolidation, it by itself is not sufficient to give the benefits of the cloud. To achieve true cloud-like elasticity in a private cloud, such that you can rapidly scale up and down in your own datacenter, will require you to allocate significant hardware capacity. While to your internal customers it may appear that they have increased efficiency, at the company level you still own all the capital expense of the IT infrastructure. Without the diversity and heterogeneity of the large number of AWS cloud customers to drive a high utilization level, it can never be a cost-effective solution.

Amazon has introduced a kind of middle ground that will allay some of the data and app security concerns while maintaining most of the capex savings, and some of the operational advantages, of using externally hosted computing services. They call it a virtual private cloud, and the idea is that Amazon buys, hosts, and cares for a set of machines that you incorporate directly into your own address space and run as part of your network of resources.

I think this is relevant to HPC readers because it begins to address one of the fundamental concerns that organizations cite when they talk about why they have to own their HPC resources. The big picture here is not one of cloud, or grid, or any of the other fads of the moment. The underlying issue is the development of a viable commercial option for enterprises to adopt HPC technologies that can be effective for their businesses without having to take the “HPC plunge” — that big leap of faith (think of it as an organizational step function that may also come along with a large outlay of cash) a non-HPC organization takes when it decides to disrupt its existing IT environment and established skills set by moving some part of its work onto a cluster of some sort.

Yes, there are probably still all of the performance issues that we’ve already talked about, but as Ian Foster and others have pointed out in comments on this site there are a variety of reasons that people turn to HPC, raw performance on a single large job being only one possible use case. Anyway we’ve already started to see companies like New Servers and Penguin (with their Penguin On Demand facility) offer “bare metal” servers configured with InfiniBand to get around some of the performance disadvantages of more highly evolved environments like Amazon’s EC2.

The development of a viable commercial smooth entry slope into HPC for the vast majority of companies, researchers, and communities not using HPC would be a Really Good Thing. The size of that market could potentially dwarf the size of the current HPC market with is mostly supported by the state (large government investments dominate the HPC business), jump starting our cash-starved industry.

If you’re interested in digging in a little further, GigaOm’s Stacey Higginbotham digs into this announcement as well, and her article is insightful.

Comments

  1. Hi John,

    I feel I need to point out that Amazon are catching up with EigenForge :) We have been offering this service since January of this year : private ‘clouds’ (for want of a better description.

    I prefer to describe it as a ring-fenced HPC system in an EigenForge data centre.

    There is no virtualisation layer and footprint on our systems so clients can enjoy the same benefits as if they were to buy the system themselves; but without the procurement woes or data centre build/upgrades cost and time delays.

    We have incorporated something further : clients do not need to commit to a 3 year term. Instead, how does 1 year sound? This way clients will effectively get a new HPC system every 12 months … if thats what they would like.

    Apologies if this sounds like a sales pitch, but Amazon does get the limelight when in this case we have been about a year ahead of them :)

    With best wishes,

    Jason Hogan-O’Neill
    —————————-
    Dr Jason J Hogan-O’Neill
    CEO and Founder

  2. Jason – absolutely not a problem. That’s the point of having a blog format and taking comments, and opposed to a one-way only publication. I’ll send you a note in a bit…I’d like to know more.

Resource Links: