Today the Square Kilometre Array (SKA) Organization announced that it is teaming up with Amazon Web Services (AWS) to use cloud computing to explore ever-increasing amounts of astronomy data. To kick things off, they just issued a Call for Proposals for AstroCompute in the Cloud, a grant program to accelerate the development of innovative tools and techniques for processing, storing and analyzing the global astronomy community’s vast amounts of astronomic data.
With the SKA, we will be generating more data than the entire Internet traffic at any single time,” said Tim Cornwell, the SKA Organisation Architect and administrator of the grant. “So we’re looking into innovative cloud solutions to help us cope with never-before-seen volumes of data, using techniques that are yet to be invented.”
AstroCompute in the Cloud grant recipients will have access to credits for AWS cloud services over a two-year period and up to one petabyte (PB) of storage for data contributed by SKA partners, which AWS will make available as a public dataset. Anyone associated with or using radio astronomical telescopes or radio astronomical data resources around the world is welcome to apply.
This is an exciting opportunity, not only for our partner institutions, but for all companies and research facilities around the world dealing with astronomy data,” said Professor Philip Diamond, SKA Organization Director-General. “The call is to help us explore how cloud computing can help process the data deluge we are expecting in astronomy in the 21st century – and in particular with the SKA.”
In its first phase of construction, SKA will include two game-changing telescopes, one consisting of more than one hundred thousand low frequency antennas, and one with about two hundred large dishes. Supercomputers will translate the enormous volume of raw data coming from the telescopes into a useable form for astronomers. With observations expected to run full-time, data will flow continuously and supercomputers will process it on the fly, transmitting useful data to an archive and deleting contaminated or otherwise unnecessary data in real time. To handle the data, and develop the know-how to process it, new smart algorithms and software will be required.
Through our Scientific Computing program, our grants and our public datasets, we’ve found that when researchers have access to the tools and data they need, they find innovative ways of solving big data challenges,” said Jamie Kinney, senior manager for scientific computing, Amazon Web Services, Inc. “The SKA is an ambitious project which presents an unprecedented opportunity to leverage a tremendous amount of data to explore the Universe.”
Beyond the field of astronomy the development of cloud processing and data analysis and visualization tools is certain to have major applications in everyday life. Supercomputing is increasingly used by pharmaceutical companies to design better drugs, by weather forecasting to refine predictions up to a week in advance, and by engineers to design smarter infrastructure.
There’s an increasingly strong link between fundamental research and computing, with all the potential spinoffs benefitting society that come with it,” said Tim Cornwell. “CERN, the European Organization for Nuclear Research, realized very early they would face a challenge to distribute the amount of data from their experiments to physicists around the world. To solve it, they created the World Wide Web. SKA is the next step.”