Frontier
Preparing for the nation’s first exascale system, the upcoming HPE Cray EX Frontier “Frontier” system at the US Department of Energy’s (DOE’s) Oak Ridge National Laboratory (ORNL), has been a colossal undertaking.
Since the spring of 2020, ORNL staff members have made numerous modifications to the building and room that will house Frontier, a system that will be capable of more than 1.5 exaflops, or 1.5 quintillion calculations per second.
But before staff at the Oak Ridge Leadership Computing Facility (OLCF) could think about staging the massive system on the floor, a lot of work needed to be done. First, they needed to dismantle the center’s old Cray XK7 Titan supercomputer. A team from HPE—Cray at that time— removed more than 430,000 pounds of components for recycling from the old “Titan” supercomputer, to make way for Frontier.
“I do not believe we have previously recycled a system as large as Titan,” said Craig Webb, a senior manager of logistics at HPE who oversees HPE’s recycling Take-Back Program.
Then, qualified electrical workers installed infrastructure to support more than 40 megawatts of power in the Frontier computer room and 8 megawatts in the center’s cooling plant. Staff in multiple divisions at ORNL helped install a new 2.5-mile-long, high-voltage power line on campus to provide the necessary power. The new power line supports 12 separate wire conductors on 84 new power poles, with approximately 160,000 feet of large wire overhead.
The team also installed nine new 4-megawatt power transformers inside the building that houses the supercomputers.
As for the room itself, staff managed to install more than 4,500 floor tiles weighing 48 pounds each—or nearly 110 tons altogether—in the space. The new floor supports more than 600 pounds per square foot and easily supports Frontier’s cabinets, which weigh 8,000 pounds each—the equivalent of a full-size pickup truck. In contrast, the cabinets that house the OLCF’s 200-petaflop Summit system weigh about 2,500 pounds each.
Construction of Frontier’s computer room. Image Credit: Carlos Jones, ORNL
To cool the new system, mechanical contractors constructed a new mechanical plant featuring a high-temperature (90°F) cooling water system with a total system volume of around 85,000 gallons of water. Four 350-horsepower pumps can each move up to 10,000 gallons of water per minute through the Frontier system. The pumps are so powerful that they could fill an Olympic-sized swimming pool in around 30 minutes. To install the system, staff connected the mechanical room’s cooling system to the data center itself with 500 linear feet of 24-inch piping. Extensive structural upgrades to the building support more than 1,000,000 pounds of overhead piping and equipment.
The new mechanical plant can cool the equivalent of 40 megawatts of computational load and can be expanded to handle 70 megawatts—the equivalent power demand of about 55,000 US homes. Frontier’s warm-water mechanical plant saves more than a whopping $1 million annually in operating costs by eliminating the need for water chillers and cold water.
All in all, more than 200 individuals supported the construction and preparation work for the project.
The research was supported by DOE’s Office of Science. UT-Battelle LLC manages ORNL for DOE’s Office of Science, the single largest supporter of basic research in the physical sciences in the United States. DOE’s Office of Science is working to address some of the most pressing challenges of our time. For more information, visit https://energy.gov/science.
source: Rachel McDowell, science writer for the Oak Ridge Leadership Computing Facility.
