By Caryn Meissner, Lawrence Livermore National Laboratory
Recognizing the importance of internal combustion engines to the nation’s energy security and a need to mitigate their negative environmental aspects, including greenhouse gas emissions, researchers are looking to develop high-efficiency, low-emission combustion engines that operate at much lower temperatures. Designing such systems requires advanced computational models that accurately represent the complex dynamical processes related to combustion, including mixing, turbulence, ignition, and flame propagation.
“Pele” is the Exascale Computing Project’s (ECP’s) application suite for high-fidelity detailed simulations of turbulent combustion in open and confined domains. The suite of C++-based codes comprises several repositories — PeleC (compressible solvers), PeleLM/LMeX (low-mach flow solvers), PelePhysics (thermodynamics, transport, and chemistry models), and PeleMP (multiphysics models) — that integrate block-structured adaptive mesh refinement (AMR) and cut-cell methods to simulate multifaceted combustion processes. Pele simulations provide the detailed physics and geometrical flexibility to evaluate the design and operational characteristics of clean, efficient, next-generation combustion technologies, including advanced ICEs for automotive, industrial, and aviation applications.
Reactivity-controlled compression ignition (RCCI) engines are seen as one of the most promising new technologies for making engines more efficient. RCCI engines use a combination of high- and low-reactivity fuels injected into a system at different times and pressures in the combustion cycle, which produces significantly less nitrogen oxide and particulate matter emissions and higher thermal efficiency. Jacqueline Chen, a senior scientist at Sandia National Laboratories’ Combustion Research Facility and the principal investigator (PI) for Pele said, “The ECP motivating problem was to perform high-fidelity simulations of the relevant processes in an RCCI engine on an exascale system. The challenge was to demonstrate the interaction of two fuels with varying reactivity under a multiphase injection strategy into an engine-relevant geometry.”
The Pele project is a prime example of the ECP objective to broaden the reach of exascale science by developing new tools that go beyond the current state of the art.
“Exascale means real capability improvement in the science we can do and how fast we can do it,” say co-PI Marc Day, who leads the High-Performance Algorithms and Complex Fluids Group within the Computational Science Center at the National Renewable Energy Laboratory. “We had to define a challenge problem that was hard and comprehensive, then we literally built up the Pele capability from scratch.” Such a feat required adapting, refining, and building upon existing codes used for other scientific research and integrating sophisticated numerical analysis techniques.
T
A PeleLMeX simulation illustrates a swirl-stabilized spray flame in a laboratory-scale aero-engine flow configuration burning sustainable aviation fuel.
he team started with several applications built for AMR and converted an existing compressive astrophysics code to one used for modeling combustion while integrating relevant chemical reactions. Pele extensively leverages AMReX, an ECP numerical library that supports scalable implementation of AMR capabilities. Chen said, “We’ve coupled AMR with cut-cell methods to simulate actual engine geometries relevant to industry and in greater detail than ever before. Pele allows us to refine the grid as the solution evolves, so we get the physics right without having to refine the entire domain. We can capture the most important physics at all the relevant scales as efficiently as possible.”
The Moonshot Realized
The Pele code was initially limited to simple configurations, but the addition of AMReX embedded boundaries allowed more complex and relevant simulations. As the Pele suite evolved, it continued to show good performance and scale even as new capability was added. A critical part of Pele’s software development chain was adapting the code to run on GPUs. Day said, “When it became clear that GPU-based systems would be the path forward for exascale, we had to rethink our CPU-based codes to make them GPU friendly. The algorithms go back decades, but we were able to pull all the pieces together in a way no one had done before and make it work.”
The ability for the team to test their code along the way allowed them to address issues as they came up, add more refinement, and iterate with applications, software, and facilities colleagues to accelerate Pele’s development. Recently, the team had the opportunity to run their code on Frontier, a milestone moment, and the results were both promising and rewarding. The code worked as expected, and they successfully demonstrated their challenge problem.
“This outcome would not have been possible without ECP. The project enabled co-design in real time with computer and domain scientists from across the Department of Energy laboratories working together and iterating through the process as challenges arose, which allowed us to advance the Pele code more efficiently,” said Chen. “It required enormous contributions in data structure and algorithm development, and the software and hardware integration experts working independently from us were essential for deploying and integrating software at the facilities and facilitating access to the machines.”
The team is now working to further enhance Pele, developing methods to produce even more complex representations of combustion chemistry and looking at ways to incorporate machine learning into representations of the models. Chen said, “We can use our detailed simulations to train machine-learning algorithms that can then be used to test our models to see how well they reproduce experimental results. In the end, we hope to have a code to exercise all algorithmic pieces developed as stretch goals and validate it on Frontier.”
Looking to the Future
Since the beginning of ECP, the impetus behind the Combustion-Pele work has evolved. Although combustion-based systems are projected to dominate the marketplace in the near term, more focus is being put on developing sustainable aviation fuels and other alternatives. Said Chen, “Over the last few years, we’ve seen increased attention to climate change and a greater interest in renewable fuels, e-fuels, hydrogen- and ammonia-based fuels and derivatives, and drop in fuels. Big drivers of combustion research include climate mitigation and greenhouse gas reductions. By changing the way we achieve combustion, we can optimize the process and make it more efficient.”
Changes in fuel also mean changes to the chemistry involved in the combustion process. Off-the-shelf codes and others used for computational fluid dynamics are more simplistic and cannot represent combustion, transport, and turbulence in a system at the detailed level Pele can provide. “Not many codes can tackle the kinds of problems we are addressing with Pele,” said Chen. “Our code is mature enough to be applied to real-world combustion problems, which have significant implications for national security, economic competitiveness, and climate change.” The code could be adapted further to look at other research areas on the horizon, such as sustainable aviation combustors, polymer upcycling, and biofuel production.
Combustion-Pele has come a long way in the seven years of ECP, gradually increasing its capability to achieve unprecedented simulations of combustion. Exascale systems such as Frontier will allow the team to further extend the reach of this new capability to access combustion kinetics, pressures, and turbulence in new regimes. Through ECP, the Pele project brought together multidisciplinary groups of people, from applied mathematicians working on the math and data structure libraries that underpin application codes to the domain scientists investigating the physics. Day said, “The successful outcome of Pele would not have been possible if we had conducted the work in the traditional stovepipe way. We didn’t do this alone; it was a team effort.”
