Over at International Science Grid This Week, Charles Harvey writes that the SKA project telescope array could someday produce data at a rate of multiple petabits per second, which is 100 times greater than current global internet traffic.
Large surveys are changing the way we process data. “Until recently, astronomers would observe at the telescope, then take their data home on disks and spend weeks or months processing it,” says Ray Norris, an astronomer at the Commonwealth Scientific and Industrial Research Organisation, Australia. “Next-generation large surveys will generate data volumes that are too large to transport, and so all processing must be done in situ”.
To meet this future need, IBM’s Center for Exascale Technology and Netherlands Institute of Radio Astronomy (ASTRON) has formed a collaboration called DOME. In development are hybrid accelerators that use pattern recognition algorithms to recognize the most useful data being produced by radio telescopes and send only that data for analysis.
Read the Full Story.