NCSA has posted additional specifications on the 10 Petaflop Blue Waters supercomputer, which is expected to come online sometime next year. The liquid-cooled system will be based on IBM’s Power7 processor, but I found the interconnect to be of particular interest:
Blue Waters will employ new IBM interconnect technology that combines concepts from IBM’s Federation supercomputing switches and InfiniBand. The high-bandwidth, low-latency interconnect will scale to hundreds of thousands of cores. The key component is the hub/switch chip. The four POWER7 chips in a compute node are connected to a hub/switch chip, which serves as an interconnect gateway as well as a switch that routes traffic between other hub chips. The system therefore requires no external network switches or routers, providing considerable savings in switching components, cabling, and power. The hub chip provides 192 GB/s to the directly connected POWER7 MCM; 336 GB/s to seven other nodes in the same drawer on copper connections; 240 GB/s to 24 nodes in the same supernode (composed of four drawers) on optical connections; 320 GB/s to other supernodes on optical connections; and 40 GB/s for general I/O, for a total of 1,128 GB/s peak bandwidth per hub chip.