It’s been nearly three years since Intel acquired Whamcloud and its Lustre engineering team. With Intel’s recent announcement that Lustre will power the 2018 Aurora supercomputer at Argonne, we took the opportunity to catch up with Brent Gorda, general manager of Intel High Performance Data Division at Intel Corporation.
insideHPC: Now that LUG 2015 is nearly over, how would you describe the state of Lustre?
Brent Gorda, GM, Intel High Performance Data Division
Brent Gorda: Once a quarter, since we started Whamcloud, I give an impromptu state of the union address internally. It’s really an opportunity for me to tell the Intel team what’s going on in the Lustre world and how we’re doing. After the last one in January, I received a number of interesting comments. “You’re more upbeat now than you’ve ever been before.” and “There’s more interesting relationships than ever.” While I have always tried to be positive, it’s really coming true now that Lustre has turned the corner, come of age, and is a real competitor in the marketplace. Certainly we all know that in high performance technical computing, it’s been the market share leader with the national labs and the universities using it. But now we are seeing traditional industries like oil and gas, aerospace, and financial services also adopting Lustre.
I wrote a State of Lustre letter a month ago that was recently published on insideHPC. One of the things I pointed out there was that PayPal has been using Lustre for a while now. They’re using it to help do real time fraud analysis on transactions that are going across the web – not just Ebay – but all sorts of transactions. This is a very interesting use-case and evidence that our favorite technology has more legs than just scientific simulation, which is where we want to take it.
insideHPC: Who is the biggest competitor against Lustre?
Brent Gorda: The one big competitor is GPFS and I think the Lustre community, by organizing together, can really go take a bite out of that business. GPFS is making a lot of money in the enterprise and that’s where we need to grow Lustre. At the LUG meeting in Denver, Intel had a number of talks where we laid out all the activities that we’re working on, bringing new enterprise features to the open source tree for this community to use to compete.
insideHPC: Who’s driving Lustre and where does Intel intend to add value?
Brent Gorda: The community is driving Lustre. Intel’s business here is adding value such as the Intel Manager for Lustre, the Hadoop coupler, the work we’ve done with the cloud, ZFS, and more. We want to be the best provider of this open source technology and outfit our partners with the most stable, highest performing and best-supported releases as we have been doing. We don’t want to control the open source technology; we think the community needs to play that role.
To that end, recently the community has voiced that it is ready to take on the tree work that this team has been doing since starting Whamcloud. We are very supportive of this and expect to play a significant role. With continued support for a single Lustre tree and hand-off to the community, we will base our work off the OpenSFS releases. We will also free up some resources as we have been dedicating 15 to 20 full time engineers to releases.
To help in this transition, we have taken a seat on OpenSFS board. We’re trying to help recruit a new chairperson for the community. I think what I want to get across here is that we are helping facilitate an alignment of the community. The community has a tremendous opportunity to make Lustre a top tier player, fueled by the growing storage market.
While we will still be contributing to the community efforts, we’re going to put that talent to work for our customers on the development projects that you’ve been hearing about here at LUG. We seek to increase the features that are necessary to compete in that market. To that end, we’re hiring for new projects and invite people to come join us.
Last week, Intel announced the Coral Project with Cray, where Argonne is buying a $200 million system as a stepping stone to Exascale. Intel Lustre is a big part of that. We have a lot of development work in there and we’re hiring for many positions here.
insideHPC: So what is Intel up to on the product side of Lustre?
Brent Gorda: I’m very excited by the progress we’ve made in developing and shipping three supported products based on Lustre. The Intel Enterprise Edition for Lustre is what you’ve heard us talk about for about two years now. This is the flagship product that features the Intel Manager for Lustre, an optimized Hadoop coupler and integration with job schedulers.
We also run in various cloud environments. AWS is the one that’s most public. I suggest you watch this space. There’s some really interesting activity going here and another clear example of Lustre moving beyond HPC into other environments.
The new product we’ve just introduced is the Foundation Edition. This is a release that will track the community release for our customers. Basically we will snapshot the community release and provide it to our customers fully tested, supported, and with maintenance releases as we’ve done in the past. Because we are no longer point on the community releases, we do not control their content. Some of our customers want to use those releases, so we need a way to support them. Foundation Edition 2.5.4 is available now to our customers and we’re pledging future releases as needed for folks who work with us.
insideHPC: How do you distribute your Intel Lustre products?
Brent Gorda: We work with over 100 hardware channel partners worldwide. You can get access to Intel Lustre from almost everyone who sells storage or compute anywhere in the world. We’re expanding this to include software partners and cloud partners. Our goal is to create the best product possible out of this open source technology and make it available economically to partners, and enable them to go after the expanding markets of HPC, cloud and Big Data.
insideHPC: Isn’t Intel a major contributor to the Linux kernel?
Brent Gorda: Yes. Intel, is the biggest contributor to the kernel this year and has been one of the top contributors consistently for the past few years. To me this shows that Intel knows how to do open source. Before we’re allowed to release any product, we need to go through a review process.
Let me explain the significance of this. Say we are planning to release a product called Intel Enterprise Edition for Lustre, with a feature in the core component of Lustre. We have to go to an open source review group and get their buy-off on the release, which includes discussion and internal guidelines about contributing the code upstream. There’s a huge amount of resistance inside Intel for anything that looks like hiding the ball with the source code. We’re not about hiding the ball. We’re all about giving you the best product, and being true to the open source community.
In my talk, I mentioned that the rules are that we must contribute the code back to the community. Many took that as me misunderstanding the GPL (which states that buyers of your product can ask for source code). What I was speaking about is our internal rules that say we will work well with the community.
insideHPC: Do you have any closing thoughts?
Yes – I want to reiterate the opportunity in front of the community. We have an open source, mature and capable technology, a growing market, and an excellent community filled with passion, pride and talent. I am very optimistic that Lustre can continue its popular position in HPC and grow to be a key player in many more areas. This is a virtuous cycle where the community will grow, vendors will increase sales, and technical people will find opportunity.
Thanks for the opportunity to chat Rich – it’s always a pleasure to see you in whatever country we happen to be in. Take care my friend.
