I think this is an interesting bit of technology (tip of the hat to HPCwire for the link). The tool, developed at TACC, is called MyCluster and allows researchers to aggregate resources on the TeraGrid such that they appear to be a single large cluster
This wasn’t a trivial task a few years ago, when Jeffrey P. Gardner, senior research scientist in high-performance computing at the University of Washington, was looking to run his research team’s large-scale cosmic simulations on the TeraGrid.
“We needed to run a humongous number of independent jobs,” Gardner recalled. “One of our users had several million computations. The advantage of having a large number of independent tasks is that you can farm them out anywhere. But the TeraGrid had no real way to do that.”
…MyCluster creates a virtual cluster, a group of linked computers that work together closely and can be managed remotely. Interacting with the user’s preferred job management system (with which most scientists are familiar), the tool sets up job proxies on whatever system can accomplish the computing tasks most quickly. When these jobs run, the provisioned computing resources appear on the user’s personal computer as a virtual cluster that dynamically shrinks or expands over time.
“All the user has to worry about is submitting his own jobs into his own personal cluster,” Walker said. “The user doesn’t have to worry about the infrastructure or deal with the heterogeneous usage policies at the different supercomputing sites.”
Cool, useful. Kudos to the development team for working on something not particularly sexy but (apparently) very useful.