The Exascale Report: On the heels of the recent FastForward program announcements, let’s take just a few steps back to frame our discussion. The FastForward program has been launched just as DARPA’s UHPC program has come to a close. Would you consider the UHPC program as being successful?
Bill Dally: I think for NVIDIA it’s been very successful. With our Echelon project, we’ve really moved the needle and made a number of very fundamental advances in technologies that are important for future HPC machines, ranging from new, low power circuit designs, to novel, very efficient energy architectures, to advances in programming languages, auto tuning, and various software development technologies. So, for us, I think there are a bunch of very tangible results that we’ve achieved under the UHPC program that are going to make a big difference in HPC in the future. One piece of evidence for that is that we’ve had 17 peer-reviewed scientific publications come out of NVIDIA’s part of the UHPC program.
The Exascale Report: Are there any lessons learned from Echelon that you will be able to directly apply to the FastForward program?
Bill Dally: I think many of them are directly applicable. We’ve made a number of advances in what we call throughput processor architecture. The architecture of processors optimized basically for a number of operations per watt. So one of those is for two-level scheduling, one of those is a very efficient register organization that reduces register fetch overhead associated with executing instructions. All of those are directly applicable to exascale computing – as are a lot of the software technologies that have been developed. We’ve made fundamental advances in scalable, parallel programming languages. There are two sort of language development prototypes under UHPC. One called Phalanx and one called Legion. Both of them have developed technologies that hopefully will be folded into future versions of CUDA and other mainstream languages.
The Exascale Report: Do you think we are taking a good approach in the way FastForward and past programs have been spread across so many different vendors?
Bill Dally: (joking) We’d prefer that they’d give all their money to NVIDIA. But I understand their desire to back more than one horse so they can hedge their bets in case something were to happen to their lead contender.
The Exascale Report: How about this two year timeframe? Is it political that we limit these programs to two years of funding, or does it make sense for some other reason?
Bill Dally: I’m not really sure where the two years comes from. You’d have to talk to someone at the DOE about that. We can make substantial progress in advancing technologies from one stage of maturity to another in a two year period. Both starting with things at the concept level and advancing them to where we have – functional prototypes and de-risk them so they can later be incorporated into products, or things at the prototype level, we can take a little bit farther and start rolling it into product plans. Now to get to an exascale is going to require maturing these technologies a lot more than two years. So our expectation is that this is the first step – that they are basically funding people to start developing the technologies that really have a long lead time to them, and that there will be a follow on program when this one ends, perhaps with a down select of some kind to take these technologies to the next level and ultimately start incorporating them into processors that will be part of pre-exascale and ultimately exascale machines.
The Exascale Report: And of course, any kind of follow on program is going to be determined by the outcome of the political election.
Bill Dally: You know, anytime government funding is involved, there’s always an element of politics in it, but I think it ought to be something that all parties can get behind – that this nation has to have a competitive HPC capability. And so I would hope that whoever wins the election – that there would be a lot of strong support for HPC and for exascale programs.
The Exascale Report: So much emphasis has been placed on FastForward announcements –obviously they came from the vendors and they (the vendors) are very proud of what they’ve accomplished – including NVIDIA, but really, these are such small allocations of research funds. Does it really make that much of a difference to a company like NVIDIA to receive an extra $12 million for a program like this? Does it really help you move exascale research forward more than you could do on your own?
Bill Dally: I think the short answer to that is yes. And to elaborate, if compared to the size of our revenues and the total expenditures we make on R&D, $12 million is perhaps small, but as an incremental funding it basically says – as we look at our list of projects that we would like to do but don’t have enough funding to do this year, and we can take over the next two years, we can take $12 million of those and accelerate them forward – do them sooner than we would have been able to do. So there are always things we want to do that we don’t have the resources to do in any given year. And because our mainstream focus is on our mainstream customers who tend to be both commercial and professional graphics customers, as well as our mainstream Tesla business which is not at exascale – it’s smaller scale HPC in the technical computing installations, what this funding allows us to do is take a number of technologies that are required for exascale that we need to do and are on our agenda, but didn’t make the funding cut for this year, or perhaps even next year, and accelerate those technologies and get to exascale a lot faster than we would on our own. So its actually making a big difference in advancing the research agenda that we would eventually get to – but we wouldn’t get to as soon without the government funding.
The Exascale Report: So Bill, just following the award announcements, within a day or two, Intel announced the acquisition of Whamcloud. Were you surprised by this announcement? To us, it’s a move that gives Intel a much bigger piece of the FastForward program ownership.
Bill Dally: I actually haven’t worried about that one way or the other a whole lot. We’re very focused on our part of the HPC ecosystem which is developing the world’s most efficient processors – basically processors that do the most FLOPS per watt, and can focus those FLOPS on HPC problems of interest to our customers. The storage system is something that – we’re happy to have somebody backing Lustre , but we don’t really care too much who it is. That’s another part of the ecosystem. We need to make sure that all the parts of the ecosystem are healthy, but right now, storage is not a high priority for us.
The Exascale Report: So, as far as HPC in general, I think it’s fair to say that NVIDIA’s name has become synonymous with HPC. And now it’s very rare to have an exascale discussion without NVIDIA’s name coming up. So what would you say FastForward really means to NVIDIA – and to the overall HPC community?
Bill Dally: What it really means is that we are taking a first step of an effort to develop exascale computing technology. I think that is very big for NVIDIA , I think that is very big for the HPC community, and I think it’s very big for the nation.
I think that High Performance Computing is critical to so many aspects of the economy. It enables people to do better designs of automobiles and aircraft; it lets them mine large amounts of data to extract information that’s important for all sorts of businesses, and many nations are investing heavily in their underlying HPC and large data infrastructure and technology base. I think that it’s very critical for the United States as a nation and for NVIDIA as a company to invest in this area, or quite frankly we’ll be left behind. I think that as a first step, and if we can continue in this direction, developing exascale technology, it positions the United States as a nation and NVIDIA as a company to be in a very competitive position to deliver computing solutions across a spectrum of exascale machines – down to departmental servers – that are at the very leading edge by the end of this decade. It requires a concerted and continuous effort in research to be able to do that. If we were not investing in programs like FastForward, we as a company and the U.S. as a nation would be left behind and somebody else would deliver those technologies in that timeframe.
The Exascale Report: Do you think we as a nation, are approaching exascale with enough of a sense of urgency? And I guess my real question is, “Is FastForward going to be fast enough?”
Bill Dally: So I would say – it’s a good start, but we need to do a lot more. Both in terms of follow on programs and also in terms of volume of funding. Quite frankly, I think that more needs to be done. We’re able to advance some of the technologies that are needed for exascale, but there are a lot more that – with $12 million – don’t make the cut. I think that – when I look at the level of investment others are making – we need to do more. I think it’s a good start and it’s good that we’re starting now rather than delaying, but I think that people need to take a really serious look at the competitive situation and realize that if the U.S. wants to be a major player in this area, they’re going to have to take it very seriously.
For related stories, visit The Exascale Report Archives.
