Supporting the Science of the Future with the New 100GbE Standard

The conversation about exascale often focuses on the computation and on building an exaFLOPS computer. But the data created by and used within those computations has to get in and out of whatever computers we end up building, and today’s networks are not up to that challenge.

In late June the IEEE Standards Association announced the ratification of the new IEEE 802.3ba standard which includes both 40Gb/s and 100Gb/s Ethernet (interestingly, the first standard to specify two Ethernet speeds). The standard is transportable over both copper and optical networks.

In a statement released by the IEEE John D’Ambrosia, chair of the IEEE P802.3ba Task Force, pointed to increasing demands inside the datacenter brought on by the growing adoption of media-rich social networking applications. “As mass-market access to these technologies continues accelerating, coupled with today’s progressively more powerful server architectures, data centers, network providers and end-users alike are finding themselves confronted by pressing bandwidth bottlenecks,” he said. “IEEE 802.3ba will eliminate these bottlenecks by providing a robust, scalable architecture for meeting current bandwidth requirements and laying a solid foundation for future Ethernet speed increases.” Companies from Google to AT&T anticipate that the technology will be influential in commercial datacenter and network design.

Of course our community’s interest in these technologies is their role in moving extreme datasets, both from one-of-a-kind instruments like the Large Hadron Collider and the Square Kilometre Array (SKA) telescope, and within HPC centers. The US Department of Energy is one of the few research organizations already testing 100 GbE routers.

According to Inder Monga, a network engineer with the DOE’s ESnet (Energy Sciences Network), ESnet is the infrastructure that the Office of Science is depending upon to meet the demands of current and next-generation instruments and scientific collaboration. He points out that the Advanced Networking Initiative (ANI) testbed in the DOE is testing 100Gbps technologies as part of a larger effort to build a cross-country 100Gbps prototype network linking DOE supercomputer sites and the international network exchange, MANLAN, in Manhattan, NY.

Steve Cotter, department head for ESnet at Lawrence Berkeley National Laboratory said “As the science community looks at collaboratively solving hard research problems to positively impact the lives of billions of people…leveraging petascale data and information exchange is essential. To accomplish this, high-bandwidth networking is necessary for distributed exascale computing. Lawrence Berkeley National Laboratory is excited to leverage this standard to build a 100G nationwide prototype network as part of ESnet’s participation in the DOE Office of Science Advanced Networking Initiative.”