The following special feature appears in the Print ‘n Fly Guide to SC14 in New Orleans, a downloadable in-flight magazine custom tailored for your journey to the Big Easy at SC14.
The Gordon Bell Prize is one of the highest honors in high performance computing. For 2014, a collaborative research project co-led by Michael Bader from Technische Universität München (TUM) and Christian Pelties from Ludwig-Maximilians-Universität München (LMU), both from Germany, and Alexander Heinecke of Intel, has been nominated for their groundbreaking code modernization work with SeisSol, a scientific software package that provides numerical simulation of seismic wave phenomena.
The team was nominated for its paper, “Petascale High Order Dynamic Rupture Earthquake Simulations on Heterogeneous Supercomputers,” a breakthrough research project that is furthering understanding of earthquakes by using numerical simulation of the propagation of seismic waves helps to understand complicated wave phenomena.
Spurring the research is the need to understand and, if possible, anticipate consequences of earthquakes, which not only strike without warning but have disastrous aftereffects. A recent reminder was the quake that shook the San Francisco Bay area on August 24, 2014 – a 6.0-magnitude earthquake whose epicenter was Napa, CA, caused $300 million in damages, left the downtown area in a shambles and killed one person. The overriding question addressed by the SeisSol team: What can be done to help prepare for earthquakes and to minimize aftereffects?
Improving accuracy of earthquake ground motion estimates based on large-scale numerical earthquake rupture simulations is one of the most important earth sciences’ “Grand Challenges.” Designs of buildings able to withstand earthquakes have made great strides in protecting lives and property, but much more needs to be done. The problem is that earthquake faulting is a complex process occurring on multiple scales and at depths that cannot be probed directly. The complete simulation of earthquake dynamics involves the physics-based computation of a fault slip, the subsequent seismic wave propagation and the resulting ground motions.
According to Prof. Dr. Michael Bader (TUM), “By generating predictions of earthquake behavior under controllable but realistic conditions, applied earthquake research impacts the design of seismic hazard mitigation measures, as well as the risk management of industrial assets. Better understanding of earthquake faulting and the simulation of future hypothetical seismic events enables improved preparation and response plans for the next ‘Big One.’ ”
Simulating large earthquake events involves multi-physics scenarios at high frequencies. Simulating these in realistic 3D Earth models requires peta- or even exascale computing power. In the SeisSol Team’s simulation of the 1992 Mw7.2 Landers Earthquake, wave propagation is solved simultaneously with earthquake faulting in a multi-physics manner.
Simulating such earthquake processes requires enormous computing power encompassing a broad array of factors, accounting for a wide variety of contingencies which scientists call a “heterogeneous solver structure.” Only supercomputers are capable of taking on problems of this scale and many of those are today based on heterogeneous supercomputing architectures comprised of a vast array of coprocessing and processing hardware.
Hence, the team utilized HPC systems enhanced with Intel® Xeon Phi™ coprocessors, simulating wave propagation on the coprocessors, while the complex faulting process is mastered by standard Intel® Xeon® processors, with the combined results providing a unified, holistic, effective solution.
Adding support for CPU accelerators or coprocessors often requires major changes to software code, a process called code modernization that enables software – originally designed for previous-generation HPC systems – to leverage the power of parallel supercomputing. The SeisSol team utilized the Intel® Language Extensions for Offloading (LEO), including pragma directives for (a) synchronous data transfers and program execution on the Intel Xeon Phi coprocessor. Since the Intel Xeon Phi coprocessor is able to execute regular C/C++/Fortran code, which can be parallelized using the widely used OpenMP standard, no major changes were required on the kernel implementation side of SeisSol.
According to Intel’s Heinecke, SeisSol employs a high-order ADER-DG (Arbitrary high order Derivatives – Discontinous Galerkin) Finite Element scheme. The ADER-DG method comes by design with a very low-communication footprint (face neighbors only) combined with a high computational intensity, making it well suited for parallel supercomputers. Additional numerics allowed the team to introduce well-separated math kernels into the computational heart of SeisSol. The time, volume and boundary kernel represent element-local and element-element operators respectively, and thus allow for overlapping communication and computation.
In addition, the most time-consuming part of each kernel can be expressed via dense and sparse matrix multiplications, which the team strongly optimized, exploiting the Intel Xeon Phi coprocessor’s close relationship to standard high performance CPUs — since highly-efficient and tuned matrix operations for Intel Xeon processors already existed, porting them to the Intel Xeon Phi coprocessor’s even more capable instruction set was a logical step and a wise investment, according to Heinecke.
A major focus of the team was to drive the size of the seismic wave propagation simulation into new regions. The goal was to achieve unprecedented resolution of earthquake faulting and synthetic ground shaking in the engineering frequency band. This high scalability is due to avoiding complicated or even global communication and focusing on direct neighbor data exchanges.
The result: one of the biggest adaptive mesh-based simulations ever conducted.
The team’s work has already been awarded the PRACE ISC Award 2014 for a scalability study of solving seismic wave propagation on CPU-based systems at 1 PFLOPS. In the case of the Gordon Bell nomination, an actual earthquake, with multi-physics components on heterogonous clusters, is solved at multiple PFLOPS.
It is a great honor being nominated for the Gordon Bell prize,” said Prof. Michael Bader, TMU. “There are many hard problems out there which require petascale or even exascale computing power. Solving these complex problems will result in major benefits for society. We would be very pleased if our results and methods helped others to improve their codes and/or models to achieve these solution results faster and with greater accuracy.”
Code modernization is a major component of the SeisSol team’s achievement.
We see a strong and continuing trend that higher code performance is no longer delivered from just replacing the existing hardware with the newest available generation,” said Bader. ”Software must also be optimized for the new hardware and its advanced features.”
He added that emerging architectures, such as the third generation Intel Xeon processors and future Intel Xeon Phi coprocessor products, will continue this trend, moving toward a convergence of available features. The key is to moving all major simulation codes toward leveraging new hardware capabilities.
How respective speedups will be achieved will depend on the individual code structures and problems being solved,” said Heinecke. “On the one hand, that may require tuning of the existing implementation. On the other, it might also mean switching to an entirely different set of algorithms that fit the hardware better.”
“By combining different techniques and different routes, the modernized code might run several factors faster on current hardware than the original implementations,” he said. “This is just the beginning of an exciting journey in modern parallel supercomputing.”
Looking for more great content like this? The Print ‘n Fly Guide to SC14 New Orleans is designed to be an in-flight magazine custom tailored for your journey to the Big Easy at SC14. Sponsored by Intel, the Guide will feature articles on code modernization.
Table of Contents: Print ‘n Fly Guide to SC14 New Orleans
- Entering the Era of Code Modernization – Welcome Letter from Raj Hazra, Intel Vice President, Data Center Group & General Manager, Technical Computing Group
- Celebrating Year One – the Intel Parallel Computing Center Program’s First Anniversary: A Q&A with Bob Burroughs, Intel Director of Technical Computing Ecosystem Enabling
- Intel Xeon Phi Takes DownUnder GeoSolutions to New Depths
- This is one Show You Won’t Want to Miss: The Parallel Universe Computing Challenge Returns to SC14
- Meet Me at the Hub. The Intel Community Hub – new for SC14
- SC14 Intel Booth Guide
- City Guide: Safety and Transportation Around New Orleans
- Local’s Guide to New Orleans Cuisine
- Leveraging Parallelism on Intel Xeon Phi Coprocessors and Multicore Processor
- Seismic Code Modernization Yields Petascale Performance and Gordon Bell Award Nomination
The Print ‘n Fly Guide to SC14 New Orleans is now available for download.
As a supplement to the guide, we also offer this Sci-Fi Original story by Rich Brueckner: Angels of Silence. We hope you can enjoy it on your flight to New Orleans.
