Sign up for our newsletter and get the latest HPC news and analysis.
Send me information from insideHPC:


BlueTides on Blue Waters: The First Galaxies

In this special guest feature, Dr. Stephen Perrenod writes that astrophysicists have completed one of the largest cosmological simulations ever on the Blue Waters supercomputer at NCSA.

The largest high-redshift cosmological simulation of galaxy formation ever has been recently completed by a group of astrophysicists (Drs. Feng, Di-Matteo, Croft, Bird, and Battaglia) from the U.S. and the U.K. This tour-de-force simulation was performed on the Blue Waters Cray XE/XK system at NCSA and employed 648,000 cores. They utilized approximately 700 billion particles (!) to represent dark matter and ordinary matter and to create virtual galaxies inside the supercomputer. The authors, who represent Carnegie Mellon University, UC Berkeley, Princeton University, and the University of Sussex, have given their simulation the moniker BlueTides.

Galaxies are the fundamental building blocks of the large-scale structure of the universe. Very early on, before the first galaxies formed, the universe was a highly uniform mix of dark matter and ordinary matter, but with about 5 times as much dark matter by mass relative to the ordinary matter (protons, neutrons, electrons) that makes up the visible parts of galaxies, including stars, gas and dust.

Figure 1 from The BlueTides Simulation paper; reproduced with permission;  z = 8 refers to a redshift of 8 when the universe was a little over 1/2 billion years old.

Figure 1 from The BlueTides Simulation paper; reproduced with permission;
z = 8 refers to a redshift of 8 when the universe was a little over 1/2 billion years old.

Galaxy formation and evolution is a very complex process which is not well understood today.
Astrophysicists want to learn the answers to key questions including:

  • When did the first galaxies and stars form, and what were their characteristics?
  • How did our own Milky Way galaxy form and evolve?
  • How does the larger environment affect and react to galaxies in the vicinity?

Addressing this problem requires a strong mix of observation, theory and computation. The observations come from the Hubble Space Telescope as well as other ground-based and space-borne infrared and optical telescopes. Since we are interested in the early universe we want to look back to the first one or two billion years of the history of our universe, which is just under 14 billion years old at present.

The physics is very complex, but the various theoretical aspects are well understood. Cosmology, gravity, hydrodynamics, gas cooling/heating, magnetic fields, radiative transport, the formation of stars and black holes must all be included in the simulation. And both the dark matter and ordinary matter components need to be traced. The dark matter provides the gravitational scaffolding – ordinary matter falls toward those regions where there is the most dark matter. As it clumps it heats, but is able to cool through radiation, and thus it is able to clump further, producing galaxies and stars.

You can start to imagine that the simulation of all these processes is a very complex problem. Essentially astrophysicists run simulations of the galaxy formation and evolution and then compare the results to deep, high-redshift, observational catalogs. This allows them to constrain certain key physical parameters. The light from distant galaxies is redshifted due to the overall expansion of the universe, driven by the Big Bang, and the higher the redshift, the further back we are looking in time, towards the origin 13.8 billion years ago. The oldest galaxies for which we have sufficient data to derive properties are at a redshift of 10, when the universe was just about half a billion years old.

The “box” in which the authors ran their simulation was approximately 2 billion light-years on a side, so a very significant portion of the universe is represented. A smoothed particle hydrodynamics method was used, including gas cooling, the simulation of molecular hydrogen, star and black hole formation and the modeling of supernovae and active galactic nuclei (quasars, etc.) and their resultant feedback on the galaxy formation process.

The BlueTides simulation is able to match actual data for very distant galaxies with redshifts in the range of 8 to 10 (age of universe only 500 to 600 million years at the time), when compared to relevant Hubble Space Telescope observations.

For the first time, the authors were able to model the reionization of the universe with a patchy distribution. Reionization refers to the time when ultraviolet radiation from the first galaxies turned atomic hydrogen and helium into a hot plasma as the electrons were knocked out of their atomic orbitals. One of the big questions is, is this reionization due primarily to active galaxy nuclei, or to the general galaxy population at the time? They find that the ionizing radiation must escape ordinary galaxies with an efficiency of 50% or more in order to be the major source. Observations tend to favor this likelihood since reionization appears to occur over an extended period of time.

The authors made a number of enhancements to the P-Gadget3 code to limit core-to-core communication and improve scalability and efficiency. They identify halos, which are the high density regions and thus the sites of galaxy formation, with a Friends-of-Friends algorithm.

They improved the scalability of the short-range particle interactions significantly. This is threaded code that scales to over 30 threads. The use of POSIX spin-locks was key to doubling the performance over the previous implementation. A particle mesh gravity solver computes long-range gravitational forces. The authors replaced the FFTW-based solver with one based on a 2-d tile FFT library, PFFT. This improves load balancing across processors.

Memory overhead was reduced by handling the finite differencing of gravity forces in Fourier space. Communication load was further reduced by compacting the particle mesh locally with a sparse matrix compression algorithm. Collectively, these modifications eliminated the scalability issues in the particle mesh algorithm that calculates the long range gravity force.

The authors also improved the sorting routine, by implementing a histogram-based sort. This resulted in a factor of 10 speedup for the Friend-of-Friends catalogue creation. They have implemented a transparent file-level striped format for storing simulations snapshots. Each snapshot is of size 47 TB.

The BlueTides simulation represents substantial progress in our understanding of early galaxy formation and evolution and the effects of early galaxies on the intergalactic medium. It also required extensive algorithmic and code development work in order to scale to 0.6 million cores with a particle load of an amazing 0.7 trillion particles. The BlueWaters system is a unique resource for the astrophysics community, that allows the creation of virtual galaxies rapidly, in large numbers, in order to enhance our knowledge of how galaxies like our Milky Way have evolved and how the universe overall has evolved.

You can learn more at the BlueTides web site.

About the author

Stephen Perrenod

Stephen Perrenod

Stephen Perrenod is author of “Dark Matter, Dark Energy, Dark Gravity: Enabling a Universe that Supports Intelligent Life.” He holds Ph.D. and Master’s degrees in Astronomy from Harvard University and a Bachelor’s in Physics from MIT. His primary research focus was on the cosmological evolution of X-ray emitting clusters of galaxies. After several years as a postdoctoral researcher in astrophysics he moved into the high performance computing field, where he has worked for over a quarter century. He has been a frequent public speaker on HPC, Grid and Cloud computing topics. When he was active in astrophysics in the 1970s, the cosmological constant – which now is of such great significance – was usually dismissed as unlikely to have a non-zero value.

References

Yu Feng et al 2015, submitted to Mon. Not. R. Astron. Soc. “The BlueTides Simulation: First Galaxies and Reionization

Yu Feng et al 2015, submitted to Astrophysical Journal Letters, “The formation of Milky Way-mass disk galaxies in the first 500 million years of a cold dark matter universe

Resource Links: