It seems everyone in HPC is familiar with Moore’s Law. But, just in case you missed that one, Moore’s law refers to the observation made in 1965 by Intel co-founder, Gordon E. Moore, that the number of transistors on integrated circuits doubles approximately every two years.
Then there is another important, but less quoted HPC observation known as Amdahl’s law. This one is named after computer architect Gene Amdahl, and is used to determine the maximum expected speedup for a fixed-sized problem when only part of the system is improved. It is often used in parallel computing to predict the theoretical maximum speedup using multiple processors.
And finally, there is one more that is perhaps not as widely known or referenced, but extremely important and relevant for several reasons. We are referring to Gustafson’s Law. This law addresses the shortcomings of Amdahl’s law which does not fully exploit the computing power that becomes available when the number of machines increases. Gustafson’s Law instead proposes that programmers tend to scale the size of the problems to use the available equipment in order to solve problems within a practical fixed time. Therefore, if faster and more parallel systems are available, larger problems can be solved in the same time.
We’re pleased to present this Exascale Report feature interview with AMD’s John Gustafson.
The Exascale Report: What inspired you to leave Intel for AMD?
GUSTAFSON: I am by nature a bit of a ‘boat rocker’ I suppose, and I wanted to make things happen fast. AMD is a very nimble company – especially the graphics part. They can change their roadmap in a fairly short order. I can imagine getting a new idea into the product roadmap, and in as little as a year, see an impact. So that was one of the big reasons. But I don’t think people realize I’ve worked on graphics for a very long time. In fact, I’ve been working on attached processors to improve graphics since the early 1980s at Floating Point Systems.
I keep returning to this idea of an outbound accelerator that somehow really helps you do these tasks that just don’t make sense to be put into a general purpose processor. And then you find all these other exciting things you can do with them as well. It’s that niche that I keep returning to throughout my career.
The Exascale Report: No one can deny AMD’s vitally important role in pushing the HPC community forward, but why now – why is this an opportune time for you to join AMD?
GUSTAFSON: Well in addition to where things are today, you have to look at the whole history. As I pieced it together, the real brains behind the success of AMD in the 2003 to 2006 era comes from innovative technology advances such as HyperTransport and 64-bit addressing – from an AMD architect named Jim Keller. Jim Keller went to Apple and he designed the processors for the iPhone and the iPad.
Well, guess who’s back at AMD? Jim Keller. He’s just downstairs from me and we can collaborate on doing some pretty exciting stuff. So I’d say, “watch this space” if you want to see a return to the good old days – which I’d say is quite possibly in AMD’s future.
The Exascale Report: During your time at Intel, you played a key role in the vendor side of the UHPC program. How has the HPC community benefited from this program?
GUSTAFSON: Every single time DARPA or the DOE says “Let’s build a giant supercomputer!”, they set a stretch goal, and it forces people to do things they would not ordinarily do from an evolutionary standpoint. They look at the energy budget and they realize we’re never going to get there if just keep waiting for the trend lines to take us there. This one is going to be the hardest factor-of-a-thousand that we’ve ever done. In fact, it almost seems like it may not be feasible – or it would be restricted to a very small set of applications and not be as general purpose as we’d like – but that makes for a very good fight because it causes you to start from scratch on a few issues. Do we really need the conventional approach to hardware-controlled caches for example? Or should we do something much more like a GPU where you’re planning the motion of every bit – and everything counts so you don’t waste a lot of energy speculatively moving memory back and forth? And you can tell where I come down on that one. I’ve always liked the organization of pipelined or vector processors because they just don’t waste things trying to get a few more percent out of a Spec benchmark, so they’ve got a big starting advantage right there. But of course the price is that somebody has to plan where everything is going to be moved to and from, and you can’t just do the ‘La-Z-Boy’ style of programming, as David Patterson likes to call it.
The UHPC program was an important step in making progress in high performance computing. It caused a lot of people to take a look at where we are going and realize that it’s not just going to be a matter of waiting for transistors to get smaller and faster like it has been to a large degree in the past. This is going to be a much bigger revolution. It’s probably going to take a lot more programming and software changes, for one thing.
The Exascale Report: Now that the UHPC program has ended, are you paying attention to any particular aspect the FastForward program?
GUSTAFSON: Well, AMD is doing both memory and processor architecture as part of the FastForward program. The memory technology is the one I watch with the most interest. When you think about how we’ve been stalled on DRAM speed for the longest time, and it used to be, a long time ago, that supercomputers had especially fast memories – if you go back far enough when supercomputers had static RAM instead of dynamic RAM because it was so much faster. It’s been a long time since we’ve had those days because the pricing is just so prohibitive. To have a balanced system you’d be at well over a billion dollars to do anything weird with the memory.
But I think the thing that’s going to break that barrier is GPUs. And the pioneers of really fast memory are going to be the people who want to play video games. And that’s going to create enough of a commodity market that it will become affordable for supercomputers to use really fast memory again. It won’t be as cheap as Radio Shack DRAM, it’s going to be specialized, but it’s going to be finally a second tier of memory that is more expensive but gives commensurate performance.
The Exascale Report: What are your thoughts on the topic of exascale funding – and the growing consensus that the U.S. is not keeping up with the investment levels of other countries?
GUSTAFSON: I still have a fair amount of confidence in American ingenuity and targeting of the right kinds of applications to run. I think we’re doing OK. Of course we’ve been through a time of tight money for both the government and private industry, so I can understand why we haven’t been quite as aggressive as we could have been, but when I look at the technology and the innovation, I don’t think we are falling behind – yet. But we clearly have to keep an eye on it.
The Exascale Report: So should we watch for AMD to become more of a serious contender in the attached processor space?
GUSTAFSON: They really already are. When I look at their market presence, they go back and forth as to whether they are No. 1 or No. 2 in the graphics space. ATI was acquired by AMD in 2006 (see: http://www.nytimes.com/2006/07/24/technology/24cnd-semi.html?_r=0) and we’re down to really only two companies that provide high-end graphics as compared to about twelve companies doing that in the year 2000. So it’s just come down to kind of a neck-and-neck race. What I’m seeing is very strong market share and a lot of revenue from the graphics products. It’s not an underdog situation at all.
The Exascale Report: So why doesn’t AMD get more market recognition for this position?
GUSTAFSON: Well, as I’ve discovered, the company takes the path of under promising and over delivering. They are very conscious about not over-hyping or getting people overly excited, and in the long run, I think we do better that way. We’ve got a great reputation for being pretty conservative in our predictions – and it is very complicated to predict the performance of these devices. They are much more complicated than CPUs.
But another interesting thing I found is that people really want us to succeed. People really like AMD. It’s almost the way they like Apple. If you look out on the web, AMD has a lot of fans out there.
The Exascale Report: So what can we look for in terms of exascale strategy from AMD?
GUSTAFSON: Well, performance-per-watt, and that’s one area where I can play a role since our discrete graphics parts are used in the servers, not just for video tasks but also for doing serious computation. If you can solve the performance-per-watt problem, then the next challenges are building a good software model, followed by resilience and reliability, all areas where AMD will undoubtedly make key contributions.
I’m questioning that we should just be pumping out ten to the eighteenth double precision operations all the time as our goal, because in most areas of computing, there is a way to improve the quality of the computation and the validity of the results, and we need to be taking a much harder look at ‘what the heck are we computing anyway?’ and not just blindly apply finer meshes and more time steps. It’s time to think about the physics really hard and ask what is the problem we are trying to solve. Don’t just keep scaling the problems in the obvious ways.
If we do that, we may just find that we need a variety of ways to represent numbers, not just 64-bit double precision – and that ultimately will save you a lot of energy.
But, let me point out a challenge that we need to address. We are doing ever more ambitious computations with ever less numerical expertise. Very few colleges are producing the needed applied mathematicians who understand round off error. I think that’s one of the things we are doing wrong – just ignoring the rounding error and hoping that double precision takes care of everything. I find when I’m trying to hire someone that I end up bringing in people from overseas because I can’t find the U.S. graduates to fill the roles. That’s both frustrating and a little scary.
The Exascale Report: So you have brought up an important concern – the grooming of the next generation of applied mathematicians, but you also have a fairly optimistic outlook about the U.S. remaining competitive. What is it in particular that you think the U.S. is doing right – where does the country excel?
GUSTAFSON: I think we are still the innovators. I think we are still the ones who will come up with completely new directions that no one saw coming. There may be some who find a way to do something cheaper, or bigger, or a government willing to spend more money, but in terms of slap-your-forehead ‘Why didn’t I think of that?’, well, I still see most of those ideas coming out of the United States.
The HPC community is still very exciting – it’s still ‘God’s country’ – and it’s a great place to be.
For related stories, visit The Exascale Report Archives.
“AMD is a very nimble company – especially the graphics part. They can change their roadmap in a fairly short order. I can imagine getting a new idea into the product roadmap, and in as little as a year, see an impact.”
“But I think the thing that’s going to break that barrier is GPUs. And the pioneers of really fast memory are going to be the people who want to play video games. And that’s going to create enough of a commodity market that it will become affordable for supercomputers to use really fast memory again.”
“It’s time to think about the physics really hard and ask what is the problem we are trying to solve. Don’t just keep scaling the problems in the obvious ways.
If we do that, we may just find that we need a variety of ways to represent numbers, not just 64-bit double precision – and that ultimately will save you a lot of energy.”
