In this slidecast, Jason Stowe from Cycle Computing describes how the company spun up a 10,600 instance HPC cluster in 2 hours with CycleCloud on Amazon EC2. Using just one Chef 11 server and one purpose in mind, this on-the-fly cluster was used to accelerate life science research relating to a cancer target for a Big 10 Pharmaceutical company.
To tackle this problem, we decided to build software to create a CycleCloud utility supercomputer from 10,600 cloud instances, each of which was a multi-core machine! This makes this cluster the largest server-count cloud HPC environment that we know about, or has been made public to date (the former utility supercomputing leader was our 6,732 instance cluster for Schödinger from 2012). If this cluster were a physical environment, analysts said it would occupy a 12,000 sq ft data center space, costing $44 million. Instead, we created this in 2 hours, with these 10,600 hosts, used it for 9 more, at a peak cost of $549.72 per hour, and turned it off for a total cost of $4,362. Wow!”