As a meta comment, this is one of those rare instances in HPC blogging when blogs actually get used for a conversation, not just for broadcasts. Cool.

I really liked Ed Walker’s article. Of course, it is really just a commentary on the specific hardware and software configurations used by Amazon and NCSA. But certainly a useful reality check.

I’d like to add another complementary perspective. As Ed shows, the NAS benchmarks execute faster on Abe than on EC2. However, what if one simply wants to run a NAS benchmark as soon as possible? In that case, the relevant metric is elapsed time from submission to the completion of execution.

Let’s say we want to run the LU benchmark, which takes ~20 secs on Abe and ~100 secs on EC2. Now let’s add in queue and startup time.

On EC2, it takes ~5 minutes to startup 32 nodes (depending on image size), so with high probability we will finish the LU benchmark within 100 + 300 = 400 secs.

On Abe, we can use the QBETS queue time estimation service to get a bound on the queue time. When I tried this in June, I was told that if I want 32 nodes for 20 seconds, then:
— With 25% chance we can get them within 100 secs,
— With 50 % chance within 1,000 secs
— With 85% chance within 10,000 secs

So, if I had to bet, I would have to go for EC2 as the “fastest” place to run the NAS LU benchmark!

Of course this result reflects not the performance of the Abe vs. EC2 schedulers, but the specific scheduling policies (and loads) that they are subject to. Nevertheless, it does provide another useful perspective on the relative merits of commercial infrastructure-as-a-service providers vs. supercomputer centers.

For more on this, see: http://www.slideshare.net/ianfoster/computing-outside-the-box-june-2009.

Regards — Ian.