Unlocking Cosmological Secrets at Durham University

Print Friendly, PDF & Email

Drawing on the power of a supercomputer from Dell Technologies and AMD, Durham University and DiRAC scientists are expanding our understanding of the universe and its origins.

In scientific circles, everyone knows that more computing power can lead to bigger discoveries in less time. This is the case at Durham University in the U.K., where researchers are unlocking insights into our universe with powerful high performance computing clusters from Dell Technologies.

What is the universe? What is it made of? What is dark matter? What is dark energy? These are the types of questions explored by the distinguished researchers using the DiRAC Memory Intensive facility hosted by Durham University’s Institute for Computational Cosmology. This level of research requires massive amounts of computational power, and Durham University gets exactly that in its legendary COSMA supercomputer clusters.

The name COSMA, which is short for Cosmology Machine, reflects the core mission of this evolving system — researching very big cosmological questions.

A purpose-built machine

Durham University rolled out the latest iteration of its COSMA supercomputer, known as COSMA8, in September 2020. In the course of developing the specifications for COSMA8, Dr. Basden, technical manager of COSMA, was particularly interested in the AMD EPYC processor, due to its higher core density and its increased memory bandwidth.

“We got access to the Dell Technologies HPC & AI Innovation Lab in Austin so that we could do benchmarking on these processors,” Dr. Basden says. A large fraction of COSMA’s simulation work is performed with software called SWIFT (SPH With Inter-dependent Fine-grained Tasking). Dr. Basden’s team and DiRAC Research Software Engineers used sample cosmology datasets with SWIFT to test the processors.

“It ran as we would hope,” he says, noting that tests were performed against other processors. “The performance of a single core was on par with other solutions, and when you’ve got more cluster cores, it’s a no-brainer. The extra core count meant the AMD EPYC processors were significantly faster,” Dr. Basden says.

For COSMA8, Dr. Basden opted for dual 280-watt AMD EPYC 7H12 processors per node with a 2.6GHz base clock frequency and 64 cores, with four server nodes in a Dell EMC PowerEdge C-series chassis with a 2U form factor for up to 512 processing cores, up to 3200MT/s memory speed to reduce latency, and PCIe Gen 4 to transfer data faster.

“We wanted a large number of cores per node, because then it meant we could cut down on the amount of internode communication,” Dr. Basden explains. “But because there are parts of the code that don’t parallelize 100 percent, we also wanted high clock rates so that the less parallelisable parts of the code stood up well, which meant the 7H12 processor would be the best option.”

Unlocking the secrets of the universe

With its rich mix of technologies, COSMA8 delivers the robust high performance compute and storage resources that scientists require when they are trying to unlock the secrets of a very big and complex universe. This work involves enormous amounts of data that must be processed at high speeds. A single simulation can produce hundreds of terabytes of data.

“About 75 percent of the universe is made up of dark matter that we don’t understand,” Dr. Basden explains. “By running these simulations, we are able to find out more about it. And, of course, when we do that, we begin to understand more about what the universe is made from.”

Ultimately, COSMA8 will enable a much greater level of discovery in comparison to simulations run on earlier generations of the COSMA supercomputer. The increased core density, larger amounts of faster DRAM, and faster PCI Express 4.0 connectivity together add up to a potent boost in performance.

“It will mean we can run massively more detailed simulations, which we can compare much better with observations from telescopes,” Dr. Basden explains. “This will help us to understand the meaning of the universe, dark matter, dark energy and how the universe was formed. It’s really going to help us drill down to a fundamental understanding of the world that we live in.”

For a closer look at the world-class supercomputing resources at Durham University, visit COSMA and read Unlocking Cosmological Secrets.