“At Cray, we are a big user and investor in Lustre. Because Lustre is such a great fit for HPC, we deploy it with almost all of our systems. We even sell and deliver Lustre storage independent of Cray compute systems. But Lustre is not (yet) the perfect solution for distributed and parallel-I/O, so Cray invests a lot of time and resources into improving, testing, and honing it. We collaborate with the open-source Lustre community on those enhancements and development. In fact, Cray is a leader in the Lustre community through our involvement in OpenSFS.”
Over at Admin HPC, Intel’s Jeff Layton writes that understanding how data makes its way from the application to storage devices is key to understanding how I/O works and that monitoring the lowest level of the I/O stack, the block driver, is a crucial part of this overall understanding of I/O patterns.
Achieving good performance on any system requires balancing many competing factors. More than just minimizing communication (or floating point or memory motion), for high end systems the goal is to achieve the lowest cost solution. And while cost is typically considered in terms of time to solution, other metrics, including total energy consumed, are likely to be important in the future. Making effective use of the next generations of extreme scale systems requires rethinking the algorithms, the programming models, and the development process. This talk will discuss these challenges and argue that performance modeling, combined with a more dynamic and adaptive style of programming, will be necessary for extreme scale systems.
“Until Docker came around some people used containers but the use was very much restricted to large organizations, like Google, that had specialized teams and training. But the containers weren’t portable between different environments. With Docker we’ve made containers easy for everybody to use; we’ve made them portable between environments; we made them exceptionally lightweight and we built up a huge ecosystem around that.”
As an example of what you can do with key-value storage and how simple it can be, Seagate has created a new storage drive called Kinetic that you address using REST-like commands such as get, put, and delete. A simple open-source library allows you to then develop IO libraries so that applications can perform IO to/from the drives. Some object storage solutions such as Swift have already been ported to use the Kinetic drives. Ceph is also developing a version that can use Kinetic drives. Other object based storage systems such as Lustre and Gluster could theoretically use this technology as well.”
In this video, Intersect360 Research CEO Addison Snell describes the dynamics unfolding in the technical computing market, and its convergence with big data analytics. “In their customer surveys on big data, Intersect360 finds that end users need to scale their environments from desktops to servers, to clusters of servers, to large supercomputers. Mr. Snell notes that IBM is a leader in HPC and Big Data that is making the investments to extend that leadership.”
In this special guest feature, Ferhat Hatay from Fujitsu writes that supercomputing technologies developed for data-intensive scientific computing can be a powerful tool for taking on the challenges of Big Data. We all feel it, data use and growth is explosive. Individuals and businesses are consuming — and generating — more data every day. The […]
“Moore’s Law got deflected in 2004, when it became no longer practical to ramp up the clock speed of CPUs to improve performance. So the chip industry improved CPU performance by adding more processors to a chip in concert with miniaturization. This was extra power, but you could not leverage it easily without building parallel software. Virtual machines could use multicore chips for server consolidation of light workloads, but large workloads needed parallel architectures to exploit the power. So, the software industry and the hardware industry moved towards exploiting parallelism in ways they had not previously done. This is the motive force behind the Big Data.”
“The failure of one parallel language — even a high-profile, well-funded, government-backed one — does not dictate the failure of all future attempts any more than early failures in flight or space travel implied that those milestones were impossible. As I’ve written elsewhere, I believe that there are a multitude of reasons that HPF failed to be broadly adopted. In designing Chapel, we’ve forged our strategy with those factors in mind, along with lessons learned from other successful and unsuccessful languages. Past failures are not a reason to give up; rather, they provide us with a wealth of experience to learn from and improve upon.”