In 2017 wind generated 6.3% of the United States’s electricity, according to the US Energy Information Administration. If the nation can use its abundant wind resources to generate 30% of its electric power, the societal and economic impact will be profound. US energy security will be reinforced by the greater diversity in the energy supply. Cost-competitive electricity could be provided to key regions of the country, greenhouse-gas emissions reduced, and the quantity of water required for thermo-electric power generation lessened.
A key challenge for wide-scale deployment of wind power without subsidy is plant-level inefficiencies. Plant-level performance losses can be as high as 20–30% due to complex terrain, unique atmospheric flow phenomena, and the complex flow interactions that occur in large wind farms, which comprise multiple arrays with significant turbine-turbine wake interactions. Addressing the challenge in reducing plant-level losses requires more knowledge of their dynamics to inform optimization of existing plants (through, for example, new control strategies), optimized layout of new plants, and the creation of new wind turbine technology.
The U.S. Department of Energy’s (DOE) Exascale Computing Project (ECP), a joint collaboration of two DOE sponsoring organizations, the Office of Science (DOE-SC) and the National Nuclear Security Administration (NNSA), has funded a research effort named ExaWind under its Application Development focus area.
ECP’s ExaWind project aims to advance the fundamental comprehension of whole wind plant performance by examining wake formation, the impacts of complex terrain, and the effects of turbine-turbine wake interactions. When validated by targeted experiments, the predictive physics-based high-fidelity computational models at the center of the ExaWind project, and the new knowledge derived from their solutions, provide an effective path to optimizing wind plants.
The ExaWind team performed large-eddy simulations of the National Renewable Energy Laboratory (NREL) 5 MW reference turbine on the National Energy Research Scientific Computing Center (NERSC) Cori Haswell system with multiple resolutions, including a model with 6 billion grid points. … This simulation capability establishes a key baseline for model and solver-stack improvements.
Large-eddy simulation (LES) is a well-known mathematical computational fluid dynamics (CFD) approach to capturing the turbulent flow structures in engineering applications. The ExaWind team recently performed a LES of multiple revolutions of the National Renewable Energy Laboratory (NREL) 5 MW reference turbine with the open-source CFD code called Nalu-Wind. The NREL 5 MW reference turbine is a notional turbine fully defined in the open domain that has the key features of large modern wind turbines.
Geometry-resolved large-eddy simulation of the NREL 5 MW reference wind turbine, which shows velocity isosurfaces at 5.5 m/s after three rotor revolutions. The simulation was performed on the NERSC Cori system with Nalu-Wind, an open-source unstructured grid, low-Mach-number computational fluid dynamics code.
A body-fitted mesh—or computational geometric representation—of the blades, nacelle (the wind turbine’s encased generating components), and tower geometry were created. The modeling applied in this effort will be used to better understand weak and strong scaling performance of the ExaWind incompressible-flow-solver software stack and to advance the movement toward next-generation computing architectures. The software stack consists of software libraries for setting up and solving the system of equations, whose solution constitutes a numerical approximation of the physical-system continuum solution. This recent success of ExaWind establishes a new baseline capability for simulating modern turbines with body-resolved meshes and enables scientists and engineers to begin understanding the complex flow physics in multi-turbine wind farms that will take advantage of future ExaWind modeling and simulation capability.
Research Context and Objectives
Current methods for modeling wind energy cost and performance fall short due to insufficient model fidelity and inadequate treatment of fundamental phenomena such as atmospheric inflow structure, turbine wake development, and subsequent turbine-turbine wake flow interactions, which are persistently significant factors, especially in complex terrain. Most design and analysis tools simplify the flow physics with empirical representations that fail to capture first principles and/or are not computationally possible with existing simulation capabilities. Among the common deficiencies of the current methods are an inability to accurately model wake structure, a lack of understanding of the impact of different atmospheric turbulent conditions, and inadequate numerical schemes for transition from mesoscale (numerical weather prediction) to LES in CFD for flow within wind farms.
The impediment to predicting and minimizing energy losses and creating new technology options that will maximize performance has compelled ExaWind researchers to develop a predictive simulation capability that the team will use to simulate a wind plant composed of more than 100 multi-megawatt-scale wind turbines located within 100 square kilometers of complex terrain. These simulations require hundreds of billions of grid points to adequately resolve the flow physics and dynamic interactions.
The primary modeling and simulation environment of ExaWind is Nalu-Wind, which is based on the Nalu code developed at Sandia National Laboratories (SNL). Nalu-Wind is an unstructured-grid code that solves the acoustically incompressible Navier-Stokes equations; these equations are well suited for solving the low-Mach-number aerodynamics around the complex moving geometry of wind turbines and wind farms. Nalu-Wind is built on the Trilinos Sierra Toolkit (STK) library and can employ either the Hypre or Trilinos linear-system solver stacks. The ultimate objective of the ExaWind project is to create a predictive wind simulation capability that will run on an exascale-class computer by 2022.
Recent Successes
The ExaWind team performed preliminary LES of the NREL 5 MW reference turbine on the NERSC Cori Haswell system with multiple resolutions, including a model with 6 billion grid points. Using a coarse mesh, a simulation of multiple turbine revolutions was demonstrated. The model exercised a sliding-mesh interface that enables rotation of the fluid mesh surrounding the turbine rotor within a fixed background mesh. These simulations have benefited from recent improvements to the strong scaling performance of the Trilinos solver stack. This simulation capability establishes a key baseline for model and solver-stack improvements.
In collaboration with the Lawrence Livermore National Laboratory (LLNL) Hypre development team, ExaWind researchers developed a novel solver algorithm that reduces the global communication requirement during the solution procedure. For ExaWind simulations, the communication overhead during linear solvers for large problem sizes is a significant challenge to improving time-to-solutions. The new Hypre solver algorithm was shown to improve time-to-solution on a large number of parallel computing message-passing interface ranks as well as on CPU-GPU architectures, for which communication requires data transfers between GPU and CPU.
Collaborations
The ExaWind project is a close collaboration between NREL, SNL, Oak Ridge National Laboratory (ORNL), and the University of Texas at Austin (UT Austin). The ExaWind CFD code, Nalu-Wind, has multiple connections in the ECP portfolio. As described above, Nalu-Wind is built on the Trilinos STK library and can employ either the Hypre or Trilinoslinear-system solver stacks. Additionally, Nalu-Wind uses the Kokkos ecosystem (Kokkos Core and Kokkos Kernels) to achieve performance portability. Other project collaborations include the open-sourse oftware/libraries TIOGA, Spack, VTK-m, and xSDK. TIOGA overset-mesh capabilities will enable Nalu-Wind to simulate multiple moving components (blade pitch and nacelle yaw, for example). The ExaWind team uses, contributes to, and collaborates with, the Spack package management project as a portable way to build and manage Nalu-Wind’s complex software stack across several computing facilities and developer laptops. The ExaWind VTK-m collaboration is helping drive extreme-scale computing by providing new in situ visualization and analysis capabilities to the ExaWind project and an independent applied project for the VTK-m team to evaluate their pipelines. The xSDK team and the ExaWind team have been collaborating in demonstrating end-to-end use of xSDK.
The ExaWind project collaborates closely with the Atmosphere to electrons High Fidelity Modeling (HFM) project, which is funded by the Wind Energy Technologies Office (US Department of Energy Office of Energy Efficiency and Renewable Energy). Whereas ExaWind is largely focused on performance and developing/optimizing the simulation software stack for exascale-class simulations, the HFM project is directed at the physics-based models, the community modeling environment, and model validation. The HFM and ExaWind partnership is critical to achieving predictive exascale simulations of wind farms.
Next Steps and Challenges
With the NREL 5 MW turbine model now available, the ExaWind team will perform additional strong and weak scaling studies of the Hypre and Trilinos solver stacks. Simulations will be done on the NERSC Cori supercomputer, as well as on the Argonne Leadership Computing Facility Mira system under the team’s 2018 Innovative and Novel Computational Impact on Theory and Experiment (INCITE) program award, “Towards Predictive Exascale Wind Farm Simulations.” The ExaWind team will also be preparing the Nalu-Wind solver stack for GPU-based simulations; exploratory simulations will be run on Oak Ridge Leadership Computing Facility SummitDev through an ECP computing allocation.
The team will enhance the Nalu-Wind code with two critical physics models. First, hybrid Reynolds-averaged Navier-Stokes (RANS) and LES turbulence modeling will be implemented, with unsteady RANS models used near blade surfaces, and LES in the wake region. Second, full fluid structure interaction will be enabled through coupling to the OpenFAST whole-turbine simulation tool, which includes geometrically nonlinear structural models for large blade deflections.
ExaWind faces a significant challenge due to the need to have moving meshes in the simulation (for the many moving components of a wind turbine). Because the connectivity between grid points changes at every time step, the discrete systems (matrices) need to be reconstructed at every time step, as do the preconditioners. These are computationally expensive operations —they can consume over 50% of the time required for linear algebra applications, for example—and can thus be a barrier to efficient simulations and scaling performance. In nonmoving grid simulations, those reconstruction/setup costs are amortized over the simulation and hence are not necessarily a critical cost in such applications.
