The Sun HPC Watercooler points us to a blog post by Dan Templeton explaining all the new goodies associated with the latest Sun Grid Engine [version 6.2 Update 5] release. More batch goodness, right? Wrong. With this release, SGE fans get to claim the first spot at the microphone with direct support for Apache Hadoop applications.
SGE 6.2u5 gets to claim the title of first workload manager with direct support for Apache Hadoop applications. What does that mean? First, it means that you can submit Hadoop applications to an SGE cluster just like you would any other parallel job. The cluster will take care of setting up the Hadoop jobtracker and tasktrackers for you. Second, it means that the SGE scheduler knows about the HDFS data locality such that it can route Hadoop jobs to nodes where the jobs’ data already lives. The net result is that you can now realistically consolidate your Hadoop cluster into your SGE cluster, saving you time, money, and lots of headaches.
What if I don’t use Hadoop? They’ve also extended their topology-aware functionality. Previously, the interface to defining topology-aware scheduler domains was somewhat cumbersome. They have no given the ability to specify topological details at job submission. This allows users to define their preferred hardware runtime environment [cores/socket; cores/node] based upon their application profile.
- Slotwise Subordination: “With SGE 6.2u5, we’re introducing finer grained preemption.”
- Extended SGE Inspect: “With SGE 6.2u5, we’ve added the ability to manage parallel environments. Going forward we will continue adding management functionality.”
- Improved Cloud Connectivity: “With SGE 6.2u3, we added the ability through the Service Domain Manager component to automatically provision additional cluster nodes from Amazon EC2 during peak periods. With SGE 6.2u5, we’ve expanded that functionality a bit and made it easier to use.”
- Improved Power Management: “We introduced the ability to automatically power down idle or underused nodes with SGE 6.u3 through the Service Domain Manager component. With SGE 6.2u5, we’ve fleshed it out a bit more and more it easier to use.”
Great stuff for the SGE user crowd. Its amazing how much a well-tuned batch environment can positively influence the user experience. For more info, read DanT’s blog post [with pointers to specific documentation notes] here.