Today Univa announced that Western Digital is using the company’s Navops Launch and Univa Grid Engine cloud software for large-scale deployments on AWS. The purpose of this collaborative project was to build a cloud-scale HPC cluster on AWS to simulate key elements of upcoming designs for their next-generation hard disk drives.
Storage technology is amazingly complex, and we’re constantly pushing the limits of physics and engineering to deliver next-generation capacities and technical innovation,” said Steve Phillpott, CIO of Western Digital. “This successful collaboration with Univa and AWS shows the extreme scale, power and agility of cloud-based HPC to help us run complex simulations for future storage architecture analysis and materials science explorations. Using AWS to easily shrink simulation time from 20 days to 8 hours allows Western Digital R&D teams to explore new designs and innovations at a pace un-imaginable just a short time ago.
Continuing its legacy of product innovation, Western Digital turned to the cloud to determine how virtually unlimited scale could allow them to solve R&D and engineering challenges faster. With this in mind, they teamed up with Univa and AWS to evaluate the impact of running their electro-magnetic engineering simulations on a massive HPC cluster built on AWS using Amazon Elastic Compute Cloud (Amazon EC2) Spot Instances.
The goal was to complete the job in the smallest amount of time and at the lowest cost. As part of this record-setting collaborative effort, Western Digital ran approximately 2.5 million simulation tasks on a Spot-based cluster of a little over one million vCPUs to determine optimal device characteristics that would help improve product quality, performance, reliability and durability for next-generation HDDs. This project required complex multi-physics simulations that needed enough capacity to run deeper simulations for increasingly complex product designs. To put this in perspective, running 2.5 million tasks of this kind in an on-premises environment would take 20 days to complete.
The electro-magnetic simulations combined with the features of AWS Spot Fleet included roughly 40,000 Spot instances and more than one million vCPUs. With AWS, Univa’s highly-scalable cluster management and scheduling capabilities of Navops Launch and Univa Grid Engine were also used to coordinate cluster management and workload execution across the wide capacity of our infrastructure and keep the cluster fully utilized even under such a very high workload.
The result was an extraordinary 60x reduction in simulation time – from 20 days to 8 hours.
“We are honored to have participated in such a unique project alongside Western Digital, who is a storage infrastructure leader,” said Gary Tyreman, President and CEO of Univa. “Univa works with hundreds of enterprise organizations who are often challenged with migrating HPC applications to the cloud, as this can typically be considerably more expensive than on-premises if not properly managed. Our Navops Launch solution gives HPC administrators the ability to control which applications are placed in the cloud, while also being able to control and monitor HPC cloud consumption and spend. I am proud of the work that the Univa team did alongside AWS, as we successfully demonstrated extreme scale HPC cloud.”
