In this video from SC15, Josh Simons and Ranjit Sawant from VMware describe the HPC Virtualization is powering scientific research. With recent performance optimizations for InfiniBand, virtualization overhead has been reduced dramatically, enabling Research as a Service.
“The combination of virtualization and cloud computing provides value to both the end users and IT providers in HPC and enterprise environments. And, once created, these private clouds can be burst to a hybrid cloud to create seamless and secure extensions of the organization’s on-premise infrastructure. Performance is the key. VMware’s approach to HPC and enterprise virtualization adds a level of flexibility and agility that cannot be achieved in bare metal environments. The HPC community can realize significant benefits from adopting enterprise-capable IT solutions grounded in proven virtualization and cloud technology. And conversely, as business IT environments become increasingly compute-intensive, lessons learned by the scientists and engineers working with HPC can be transferred to their counterparts in the enterprise. It’s a win-win situation.”
insideHPC: I haven’t heard much about virtualization this week, so we’re going to line up the show with your talk here and tell us, what are you guys showcasing here this year?
Josh Simons: Yeah, sure. What we are showing are a couple of things. We’re showing performance and you and I’ve talked over the last couple of years and what we’ve been doing is driving down InfiniBand latencies and driving up performance across the board, so we’re able to actually address a wider and wider array of application types over time. We’re at the point now where in the near future, we’ll actually completely eliminate the latency overheads for InfiniBand. We’re doing testing with FDR InfiniBand, EDR should be the same, so we’re pretty excited about that.
But in addition, what we’re really working on is building an entire stack – a solution stack – that addresses a variety of HPC needs. So we have the bare virtualization platform, which has a lot of innovation, interesting capabilities in it and the good performance that we’re talking about. But we also have a private cloud that sits on top of that. And one of the big things that’s finally happening is we’re finally getting customers to understand that using virtualization in your on-prem HPC environment is actually something to seriously look at as opposed to just looking and going to private cloud. Because as you know, if you look at the size, the scale of the number of nodes that are being deployed in HPC environments and you look at the intensity that they’re being used at, replacing that on-prem resource with an off-prem – regardless of the provider – but we’re using that off-prem.
It’s just economically– and feasible to actually go do that. So HPC as a community is going to have on-prem resources for a long time. So we do a lot of work talking to customers and we have had a lot of interest in doing private cloud on-prem. Above that we actually are now building an HPC-specific solution that targets life sciences customers and customers that are less interested in dealing with IT, and more interested in focusing on the science and Rangit, on our team, is working on that.
insideHPC: Ranjit, can you tell us more about what you’re working on?
Ranjit Sawant: Sure. We have enabled virtualization for HPC but it’s important to bring the benefits of virtualization to end researchers in a way they can use it, right? So what we have done is we have created the solution plus VMware High-Performance Analytics, which allows researchers to author their own workloads, they can collaborate it, they can clone it, then they can share it with other researchers. And they can modify their workload – they can fine tune it.
Not only that, with a click of a button, they can share their data sets, they can share their recipes which are nothing but abstracted workloads. They can connect the workloads to remote data sets. The workloads are pretty lightweight as compared to the data sets, which are in petabyte. So we can move those things and connect to the data sets in the on-the-fly. Basically, what we’re doing is – the benefits of virtualization – we are making it available to researchers and their teams in a way that they can self-service the HPC workloads and don’t have to bother about IT as such and then focus on the science. All this can be done within a few clicks using our High-Performance Analytics [portal?].
insideHPC: I just want to wrap things up here. Now you’ve got performance that is comparable to bare metal, you’re saying?
Josh Simons: Right, yeah.
insideHPC: Yeah, and so when I think of virtualization, I don’t have to think slow anymore. Do I?
Josh Simons: Exactly. In fact, the best conversation I had at this show is with someone from [press?], who walked up to the– actually he walked up to the graphics here and he said, “First off, you got 15 to 20% overheads on performance, so why would I virtualize? And even if you didn’t have performance overheads, why would I virtualize?” So we took him through the performance, showed him that there’s really no overheads and then took him through the use cases, things like being able to run multiple operating systems very dynamically on your resources, have security separation between virtual machines and workloads, separation, there are a whole list of things you can do with a virtualized environment. Five minutes later, he walked away – a totally changed guy – he’s like, “I totally get it [laughter].” So it’s pretty awesome.