Sign up for our newsletter and get the latest HPC news and analysis.

Why Cray is Backing Lustre and OpenSFS

Cory Spitz, Manager, Storage and Data Management R&D File Systems at Cray

“At Cray, we are a big user and investor in Lustre. Because Lustre is such a great fit for HPC, we deploy it with almost all of our systems. We even sell and deliver Lustre storage independent of Cray compute systems. But Lustre is not (yet) the perfect solution for distributed and parallel-I/O, so Cray invests a lot of time and resources into improving, testing, and honing it. We collaborate with the open-source Lustre community on those enhancements and development. In fact, Cray is a leader in the Lustre community through our involvement in OpenSFS.”

Understanding I/O Patterns at the Block Level with ioprof

Jeff Layton

Over at Admin HPC, Intel’s Jeff Layton writes that understanding how data makes its way from the application to storage devices is key to understanding how I/O works and that monitoring the lowest level of the I/O stack, the block driver, is a crucial part of this overall understanding of I/O patterns.

Bill Gropp on Engineering for Performance in HPC

Bill Gropp

Achieving good performance on any system requires balancing many competing factors. More than just minimizing communication (or floating point or memory motion), for high end systems the goal is to achieve the lowest cost solution. And while cost is typically considered in terms of time to solution, other metrics, including total energy consumed, are likely to be important in the future. Making effective use of the next generations of extreme scale systems requires rethinking the algorithms, the programming models, and the development process. This talk will discuss these challenges and argue that performance modeling, combined with a more dynamic and adaptive style of programming, will be necessary for extreme scale systems.

Ben Golub on How Docker will Change the Datacenter

Ben Golub, CEO of Docker

“Until Docker came around some people used containers but the use was very much restricted to large organizations, like Google, that had specialized teams and training. But the containers weren’t portable between different environments. With Docker we’ve made containers easy for everybody to use; we’ve made them portable between environments; we made them exceptionally lightweight and we built up a huge ecosystem around that.”

Is Key-Value Data Storage Coming to a File System Near You?

Jeff Layton

As an example of what you can do with key-value storage and how simple it can be, Seagate has created a new storage drive called Kinetic that you address using REST-like commands such as get, put, and delete. A simple open-source library allows you to then develop IO libraries so that applications can perform IO to/from the drives. Some object storage solutions such as Swift have already been ported to use the Kinetic drives. Ceph is also developing a version that can use Kinetic drives. Other object based storage systems such as Lustre and Gluster could theoretically use this technology as well.”

Video: For Big Data and Technical Computing, Infrastructure Matters

Addison Snell, Intersect360 Research

In this video, Intersect360 Research CEO Addison Snell describes the dynamics unfolding in the technical computing market, and its convergence with big data analytics. “In their customer surveys on big data, Intersect360 finds that end users need to scale their environments from desktops to servers, to clusters of servers, to large supercomputers. Mr. Snell notes that IBM is a leader in HPC and Big Data that is making the investments to extend that leadership.”

Slidecast: Introducing the TPCx-HS Benchmark for Big Data

raghu

In this slidecast, Raghu Nambiar from Cisco introduces the new TPCx-HS Benchmark for Big Data. “TPCx-HS is the first vendor-neutral benchmark focused on big data systems – which have become a critical part of the enterprise IT ecosystem. “

Supercomputing Technologies for Big Data Challenges

Ferhat Hatay

In this special guest feature, Ferhat Hatay from Fujitsu writes that supercomputing technologies developed for data-intensive scientific computing can be a powerful tool for taking on the challenges of Big Data. We all feel it, data use and growth is explosive. Individuals and businesses are consuming — and generating — more data every day. The […]

Why Big Data is Really Big Parallelism

Robin Bloor

“Moore’s Law got deflected in 2004, when it became no longer practical to ramp up the clock speed of CPUs to improve performance. So the chip industry improved CPU performance by adding more processors to a chip in concert with miniaturization. This was extra power, but you could not leverage it easily without building parallel software. Virtual machines could use multicore chips for server consolidation of light workloads, but large workloads needed parallel architectures to exploit the power. So, the software industry and the hardware industry moved towards exploiting parallelism in ways they had not previously done. This is the motive force behind the Big Data.”

Why Chapel for Parallel Programming?

Brad Chamberlain

“The failure of one parallel language — even a high-profile, well-funded, government-backed one — does not dictate the failure of all future attempts any more than early failures in flight or space travel implied that those milestones were impossible. As I’ve written elsewhere, I believe that there are a multitude of reasons that HPF failed to be broadly adopted. In designing Chapel, we’ve forged our strategy with those factors in mind, along with lessons learned from other successful and unsuccessful languages. Past failures are not a reason to give up; rather, they provide us with a wealth of experience to learn from and improve upon.”