This is the second entry in an insideHPC series that explores the HPC transition to the cloud, and what your business needs to know about this evolution. This series, compiled in a complete Guide available here, covers cloud computing for HPC, industry examples, IaaS components, OpenStack fundamentals and more.
Cloud technologies are influencing HPC just as it is the rest of enterprise IT. The main drivers of this transformation are the reduction of cost and the increase in accessibility and availability to users within an organization.
Traditionally, HPC applications have been run on special-purpose hardware, managed by staff with specialized skills. Additionally, most HPC software stacks are rigid and distinct from other more widely adopted environments, and require a special skillset by the researchers that want to run the applications, often needing to become programmers themselves. The adoption of cloud technologies increases the productivity of your research organization by making its activities more efficient and portable. Cloud platforms such as OpenStack provide a way to collapse multiple silos into a single private cloud while making those resources more accessible through self-service portales and APIs. Using OpenStack, multiple workloads can be distributed among the resources in a granular fashion that increases overall utilization and reduces cost.
While traditional HPC systems are better for a certain workload, cloud infrastructures can accommodate many.
Another benefit of breaking down computation siloes is the ability to accommodate multidisciplinary workloads and collaboration. While traditional HPC systems are better for a certain workload, cloud infrastructures can accommodate many. For example, they can be used to teach computation techniques to students as well as provide a resource for researchers to make scientific discoveries. Traditional HPC infrastructures are great at solving a particular problem, but they are not very good at the kind of collaboration that modern research requires. A multidisciplinary cloud can make life-changing discoveries and provide a platform to deliver those discoveries to other researchers, practitioners or even directly to patients on mobile devices.
Definitions of cloud computing vary, but the National Institute of Standards and Technologies (NIST) has defined it as having the following characteristics:
- On-demand self-service
- Broad network access
- Resource pooling
- Rapid elasticity
- Measured service
Applied to HPC workloads, the service and delivery model is generally understood to include the following buckets, either individually or combined (derived from NIST definition):
- Software as a Service (SaaS) − The capability provided to the consumer is to use the provider’s applications running on a cloud infrastructure. The applications are accessible from various client devices through either a thin client interface, such as a web browser (e.g., web-based email), or a program interface. The consumer does not manage or control the underlying cloud infrastructure, including network, servers, operating systems, storage or even individual application capabilities, with the possible exception of limited user-specific application configuration settings. In terms of HPC workloads, largescale simulations can reside on the servers in the cloud and only the results are then transferred back to the client device. The HPC software can be used as a service, where input files can be uploaded, simulations run and the results returned to a local device.
[clickToTweet tweet=”Red Hat – Cloud technologies are influencing HPC just as it is the rest of enterprise IT. #cloudcomputing” quote=”Red Hat – Cloud technologies are influencing HPC just as it is the rest of enterprise IT. #cloudcomputing”]
- Platform as a Service (PaaS) − The capability provided to the consumer is to deploy onto the cloud infrastructure consumer-created or acquired applications created using programming languages, libraries, services and tools supported by the provider. The consumer does not manage or control the underlying cloud infrastructure, including network, servers, operating systems or storage, but has control over the deployed applications and possibly configuration settings for the application-hosting environment. New classes of HPC applications can be developed in the cloud, as long as the required APIs are available to the developers.
- Infrastructure as a Service (IaaS) − The capability provided to the consumer is to provision processing, storage, networks and other fundamental computing resources where the consumer is able to deploy and run arbitrary software, which can include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure but has control over operating systems, storage and deployed applications, and possibly limited control of select networking components (e.g., host firewalls). This is perhaps the most understood cloud service, where a user or team would contract with a cloud provider, rent the systems and run the HPC application.
Public clouds will contain sufficient compute servers, storage amounts and the networking necessary for many HPC applications.
The various types of infrastructure described here can physically reside or be deployed over the following types of clouds:
- Public clouds – When cloud computing is mentioned, most people think of the main cloud providers such as Amazon Web Services (AWS), Microsoft Cloud or Google Cloud Platform. These are all very large public cloud providers that can host many thousands of customers simultaneously. They can provide almost any service (IaaS, PaaS, SaaS) that a customer desires, at a scale that the customer can afford. Customers can be isolated from one another or larger systems can be assembled that allow multiple customers to collaborate as needed. Public clouds will contain sufficient compute servers, storage amounts and the networking necessary for many HPC applications. In many cases, a public cloud from the major vendors will have significantly more resources available than an on-premises data center for applications that scale. It may also be able to run many more concurrent applications from a single customer than an in-house data center.
- Private clouds – Using similar technology to a public cloud implementation, a private cloud resides within an organization’s physical infrastructure and is meant to provide services to the employees or contractors. The entire cloud infrastructure would be sized for the organization’s needs, but would be used by the entire company for a variety of IT tasks. While the total resources of a private cloud are limited, servers, storage and networking could be allocated to HPC projects as needed to meet deadlines. Alternately, resources that are used for part of the day could be repurposed for HPC applications at other times of the day.
- Hybrid clouds – In a hybrid cloud environment, the cloud infrastructure would comprise two or more different cloud infrastructures. These clouds remain separate, but can be combined to solve a certain IT problem. In many cases, one cloud may need additional capacity at a given time, and would “burst” some workloads to a distinct cloud. As deadlines approach that require simulations or other HPC applications to be run, this overflow from normal operations could be sent to another cloud as an extension of the private cloud or in-house data center. The ability to seamlessly expand computing resources for HPC applications as demand grows is a very important consideration, especially as more organizations set up cloud-computing environments.
Over the next few weeks this series on the HPC transition to the cloud will cover the following additional topics:
- The HPC Transition to the Cloud
- IaaS Componants
- OpenStack Fundamentals
- Future Directions
You can also download the complete report, “insideHPC Research Report on HPC Moves to the Cloud – What You Need to Know,” courtesy of Red Hat.
Michael, this a very lucid, crystal clear article on HPC transformation. I can see the synergy of Red Hat with Ubercloud