In this special guest feature, Linda Barney writes that researchers at the University of Cambridge are using an Intel Xeon Phi coprocessor-based supercomputer from SGI to accelerate discovery efforts.
Today, scientists at the Stephen Hawking Centre for Theoretical Cosmology (CTC) at the University of Cambridge are doing research to help refine the understanding of the formation of the early universe as it relates to the Big Bang theory and their research has drawn the attention of the global scientific community. The CTC team is using the COSMOS supercomputer, the largest shared-memory supercomputer in Europe, located at the University of Cambridge and part of the national DiRAC HPC Facility funded by the UK‘s STFC (Science and Technology Funding Council.) This site is used by a variety of researchers in cosmology, astrophysics and particle physics as well as modeling and mathematics research. In 2014, the CTC containing the COSMOS supercomputer was named as an Intel® Parallel Computing Centre (Intel® PCC.) At the time, Professor Stephen Hawking, founder of the COSMOS Consortium, said: “These are exciting times for cosmology as we use COSMOS to directly test our mathematical theories against the latest observational data. Intel’s new technology and this additional support will accelerate our scientific research.” This article is about one such CTC advance applying complex new analysis methods to maps of the primordial Universe.
Mapping the age and structure of the universe
Scientists at CTC are doing research to help refine the understanding of the formation of the early universe as it relates to the Big Bang theory. In early 2015, the European Space Agency (ESA) Planck Satellite provided the highest quality full-sky maps of the universe. The CTC team is using the COSMOS supercomputer to analyze Cosmic Microwave Background (CMB) data captured by the Planck Satellite. CMB is the relic radiation or “first light” to escape once the very hot temperature from the Big Bang cooled after about 300,000 years. This is like taking a snapshot of the universe as it was over 13 billion years ago.
Previous cosmology research has focused on analyzing the relation of the CMB temperature at two points in the sky, which is known as the power spectrum. James Fergusson, CTC Lecturer states, “The Big Bang Theory was confirmed in the 1990s with the COBE experiment measuring the CMB which is one of its key predictions. Measurements of the power spectrum of the CMB by experiments like the recent Planck satellite accurately determine its age.”
What is missing from existing research
Fergusson indicates that the standard Big Bang Theory does have some issues in that it can’t explain why the temperature of the CMB is so uniform in every direction. “This can only be done by supposing Inflation, a period of rapid exponential expansion where the universe doubled in size over 80 times in a tiny fraction of a second. Inflation has been extremely successful predicting the shape of the power spectrum of the CMB and solving all the known issues with the Big Bang model. However, while we know Inflation works we do not know why it occurs or what substance could drive it. Trying to find a natural mechanism for creating inflation is one of the main goals (with determining the nature of Dark Energy and Dark Matter) of cosmology at the moment. Fundamentally, we are trying to answer the question “Where did space come from?” according to Fergusson.
CTC team pushing the boundaries of cosmology research
The CTC team wanted to push the boundaries further beyond two points and study the relationship between three points (or triangles in the sky), known as the bispectrum.
All the information we currently have about the processes present at the very beginning of our universe are contained in the statistics of the tiny density fluctuations. By going beyond the power spectrum to the bispectrum, we are opening a new window onto the origin of space itself”. Dr. James Fergusson, CTC Researcher.
The following graphic shows the bispectrum as a 3D object defined inside the tetrahedral region.
What the CTC research means
In the CTC research, a blue signal means there is a large positive correlation between the three temperatures at three points (defining a specific shaped triangle.) Wherever there is a large red signal there is a negative correlation. So what does this mean?
Fergusson explains that “The CMB is the near uniform infra-red ‘light’ that we can see coming from every direction in the universe. It is the faint afterglow of the Big Bang coming from around 380,000 years after inflation occurred and is the oldest thing we can observe. The light has tiny variations in intensity in different directions, about one part in 100,000, which are the seeds created by quantum fluctuations during inflation from which all structure in our universe has grown, from stars and planets up to giant clusters of Galaxies. It is the statistics of their fluctuations which tell us about how inflation must have worked. The two point function, or power spectrum, only tells you about the variance (the width) of a probability distribution. The three point correlator, or bispectrum, tells us about its shape, roughly its skewness (how far it leans over to one side or the other) which provides a new vantage point from which to study inflation.”
Interesting finding relating to axion particle
CTC’s analysis of the Planck satellite data confirms an important signal in the bispectrum predicted by general relativity, but it also reveals other possibilities as-yet-unexplained signals which could tell us much more about the early universe. The standard picture of inflation predicts the fluctuations to be almost perfectly Gaussian so the bispectrum plotted in the picture should be just random noise (which averages to zero). “So any region where the signal is larger than we expect is interesting in that it may point to new physics. The blue signal showing a large signal in the plot relates to models where inflation may have been driven by a new particle called an axion,” states Fergusson.
Popular axion inflation models are inspired by fundamental theory (superstrings) in which there are more than just the usual three spatial dimensions. Today these extra dimensions are “wrapped up” so tightly that we cannot observe them, but in the high energy early universe they can leave an imprint, that is, a periodic signal that might give rise to the apparent oscillations in the Planck bispectrum. Professor Paul Shellard, CTC Director says: “Searching for evidence of new physics is an exciting prospect which strongly motivates CTC research, but its remains to be seen whether these ‘hints’ of oscillations become more statistically significant next year when the Planck data improves further.”
CTC Cosmology research requires use of supercomputer
Cosmology research requires large-scale High Performance Computing (HPC) simulations and would not be possible without supercomputers and specialized software. Many millions of computer core hours were used analyzing the two point power spectrum, with the COSMOS supercomputer playing an important role in this research. CTC’s recent experiments measuring the bispectrum (triangles in the sky) required a supercomputer using the latest Intel® Xeon Phi™ coprocessors along with specialized software.
Calculating the three-point correlator is computationally challenging because averaging over all possible triangles for the 20 million Planck pixels would require 1021 complex operations—we would need the world’s biggest supercomputer for calculating the bispectrum just once.” Dr. Juha Jäykkä, Manager of the Intel® PCC at University of Cambridge.
COSMOS Supercomputer specifications
- COSMOS Supercomputer can perform 38.6 trillion calculations per second (TFLOPS)
- Based on SGI UV2000 systems with 1856 cores of Intel® Xeon® processors E5-2600
- 14.8 TB RAM and 31 Intel® Xeon Phi™ coprocessors
According to Dr. Juha Jäykkä, Manager of the Intel® PCC at University of Cambridge, “The Intel® Xeon Phi™ coprocessor allowed us to do the same work on a significantly smaller number of Intel® Xeon® processor sockets and even faster. Providing a two-fold advantage: shorter times to wait for jobs to run by using fewer sockets and shorter runtimes than we would have on Intel® Xeon® processor thanks to the superior architecture of the Intel® Xeon Phi™ coprocessor.”
Dr. Jäykkä further indicates other benefits from being recognized as an Intel® Parallel Computing Centre (Intel® PCC). “Because of our Intel® PCC status, we get the ‘inside track’ to Intel on various topics, which is invaluable for the code modernization work. The Intel® PCC meetings provide our team with a wider perspective of what code modernization and optimization is happening elsewhere and allows us to build new collaborations and networks.
Specialized software used in CTC cosmology research
The COSMOS team developed MODAL, a 3D statistical correlation code to find and interpret the very tiny fluctuations that have been measured between different CMB data points on the sky. These fluctuations can provide insight into new physics theories about how structures like stars and galaxies formed in the Universe. “The MODAL algorithms decompose the bispectrum into the physical modes we need to study and then we have optimized and modernized our pipeline very effectively to run on Intel® Xeon Phi™ coprocessor hardware. The COSMOS team optimize their MODAL code to use parallelism and modernization so that it can scale across many cores. The current generation of Intel® Xeon Phi™ coprocessor gives us more performance using less power.” states Dr. Jäykkä. The chart below illustrates a speedup of over 100 (80) times gained through combination of optimizations and modernizations, running on Intel® Xeon Phi™ coprocessor (Intel® Xeon® processor) hardware.
The CTC researchers also used the open source OSPRay ray traced volume engine which has high performance and high fidelity volume rendering processed directly from system RAM. OSPRay maintains interactive frame rates for manipulating up to four data sets simultaneously which allowed the researchers to quickly explore and fully compare theoretical models versus the observed data to provide significant insights in a way that was not previously possible.
We have managed to modernize and optimize the main workhorse code used in the research so it now runs at 1/100-1/1000 of the original runtime. This allows us to tackle problems which would have taken unfeasibly long to solve. Secondly, it has opened windows for previously unthinkable research, namely using the MODAL code in cosmological parameter search: this is a problem which is constantly being solved in an iterative process, but adding the MODAL results to the process has only become possible with the improved performance.” Dr. Juha Jäykkä, Manager of the Intel® PCC at University of Cambridge.
Future of CTC Research enabled by next-generation COSMOS supercomputer
Future research using the COSMOS supercomputer by the CTC team will address issues raised in the current bispectrum research about the nature of the Universe, especially using new cosmological data from ongoing surveys of the positions of billions of galaxies. Professor Shellard says: “CTC research focuses on the two circumstances in which the hidden basic fabric of the Universe is shaken: in the enormous densities of the Hot Big Bang and during the extreme collapse to form black holes. Computer modeling of these highly nonlinear events has become an essential research tool and so we depend on advances in HPC technology to gain new insight.”
“We are also anxiously awaiting the next generation of Intel® Xeon Phi™ coprocessor product family, as that will allow even more new research to be done such as investigating field theories in the early universe and black holes using adaptive mesh refinement (AMR). The next generation of Intel® Xeon Phi™ coprocessor product family will bring three important advantages: it will provide more computing power, it will provide a better memory subsystem (something AMR finds very useful), and it will increase performance/watt. Furthermore, the Intel® Omni-Path Architecture interconnect will be especially important for AMR because AMR load-balancing requires extensive computer communication which is not feasible with existing accelerators,” states Dr. Jäykkä.
If you attended ISC15 in Frankfurt, you got a sneak preview of new Intel technologies making astronomical calculations, such as those tackled by COSMOS, possible. In the company’s booth, a COSMOS simulation was running on a demo of the forthcoming Intel® Xeon Phi™ code name Knights Landing coprocessors supported by the first public pre-release demo of the Intel® Omni-Path Architecture.
Intel Omni-Path Architecture supplied the huge bandwidth – 100 Gbps – and low latency needed to run applications of this magnitude. The advanced fabric will be released in the fourth quarter of this year.
According to Joe Yaworski, Intel Director of Fabric Marketing for the HPC Group, Intel® Omni-Path Architecture is primarily an evolutionary product incorporating some outstanding revolutionary designs. It builds on the best features of the company’s five-year-old True Scale Fabric and adds new capabilities that will take Intel® Omni-Path Architecture into the Exascale era.
Linda Barney is the founder and owner of Barney and Associates, a technical/marketing writing, training and web design firm in Beaverton, OR.
Dr. Stephen Hawking bio: http://www.imdb.com/title/tt2830416/?ref_+nv_sr_4
Centre for Theoretical Cosmology: http://www.ctc.cam.ac.uk
http://arxiv.org/abs/1503.08809 covers the “computer science” work associated with this article, along with a chapter in the forthcoming book High Performance Parallelism Pearls v2 by Jim Jeffers and James Reinders from Intel.
For articles covering some of the science (part of High Performance Parallelism Pearls v2) and majority of the Intel® PCC work, see:
Fergusson, J. R., Gruetjen, H. F., Shellard, E. P. S., Wallisch, B.; Phys. Rev. D 91; 123506 (2015)
Fergusson, J. R., Gruetjen, H. F., Shellard, E. P. S., Liguori, M.; Phys.Rev. D91 (2015) 2; 023502 and the Planck (satellite) collaboration articles: