This sponsored post from Ian Colle, General Manager-AWS Batch and HPC, for Amazon Web Services, explores how we are always looking for resources to “do everything we want to do.” What if the number of cores were virtually unlimited?
Ian Colle from AWS
The HPC industry continues to challenge the limits of computing. We speed up the clock speed on processors and shrink their size. We grow the size of memory. We increase the capacity of storage. We push more bits across fatter networks in new architectures. We come up with new scheduling algorithms to maximize utilization of the capacity we have. And we optimize the cost of operations constantly to get budgets allocated to buy more capacity.
The cloud offers the greatest democratization of HPC resources globally.
However, one thing remains constant – we never seem to have sufficient resources to do everything we want to do. Our hunger for time on an HPC system is never satiated. And so, we create queues, and allocation policies, and teach researchers how to write grant proposals that include system time, or there’s always a hope of a chance to test out a great idea over the holidays. We often do not realize that we are, in fact, burying innovation behind the walls of bureaucracy and delays.
What if that queued genomic sequencing holds a cure for pediatric cancer? What if the delayed seismic reservoir simulation holds the answer to untapped energy resources? What if the proposed computational fluid dynamics run holds the key to making an economically viable return to supersonic flight possible? How do we place a price on delaying these innovations?
At AWS, we’ve crafted a range of HPC solutions that allow you to unbound your innovation. We combine the latest compute, networking, storage, security, cloud orchestration, and visualization technologies, with a vibrant partner and ISV community. This allows us to offer a highly customizable computing platform for organizations planning to leverage the cloud for a broad range of compute intensive workloads. These workloads span not only the traditional HPC applications like genomics, life sciences research, financial risk analysis, computer aided design, and seismic imaging, but also range to the emerging applications like machine learning, deep learning, and autonomous driving. AWS HPC solutions remove the traditional challenges associated with on-premises clusters: fixed infrastructure capacity, technology obsolescence, and high capital expenditures. AWS gives you access to virtually unlimited HPC capacity, built from the latest technologies. You can quickly migrate to newer, more powerful compute instances as soon as they are made available on AWS. This removes the risk of on-premises CPU clusters becoming obsolete or poorly utilized as needs change over time.
Our partners have demonstrated scaling clusters of over a million vCPUs on AWS. What could you do with that kind of capacity at your fingertips? How might you look at problems differently? Imagine creating your own custom cluster mix of GPUs, CPUs, storage, memory, and networking – just the way you want it, then running your experiment, getting the results, and then tearing it all down. Then a colleague comes along and dreams up a new mix of infrastructure, spins up her cluster, runs her computation, gets the solution, and shuts it all down again. Think of this process repeating across thousands of companies, hundreds of research universities, and dozens of national laboratories. What kind of a difference might that make?
As we approach the era of exascale computing it’s easy to focus on the future haves (those with access to an exascale system) and the have nots (those who do not). The cloud offers the greatest democratization of HPC resources globally. All it requires is an internet connection and a dream. What would you do with a million cores?
AWS: HPC unleashed – innovation unbound – the future reimagined.
Ian Colle is General Manager-AWS Batch and HPC, Amazon Web Services.
