Barry Bolding Interview – Interconnects

Print Friendly, PDF & Email

Click Here for the Audio Interview with Barry Bolding

The Exascale Report: Will interconnect be part of the solution or part of the problem as we move toward exascale?

Bolding: In our view, interconnects are the lifeblood of any kind of exascale system. Scalability is what we build our systems to do – it is Cray’s bread and butter – it’s our value proposition – that we actually build systems that can run applications to scale – and that means production applications – not just Linpack or just a few kernels, we build systems to run full applications to scale – and there are few interconnects – certainly there are no commodity interconnects that today run petascale applications – and so for exascale – we believe the challenge will be to provide production networks and the software that builds on top of those that will allow people to run their applications to scales that they haven’t to date been able to.

TER: So Barry, when you’re talking about interconnect, you’re looking at an entire ecosystem of facilities – not just one or two components. Is that correct?

Bolding: “Absolutely. What we’ve found – or what I think the market has proven out is that the successful systems – at scale – have been built to integrate software, hardware, interconnect technology and a number of other components in a way that allows the systems to run at scale. Building strictly from commodity components has not to date been very successful for building these large systems to scale. What tends to happen is – they’re built to scale – they run Linpack – and then they’re broken into islands. So commodity interconnects to date have really been successful at an island model – one of breaking up jobs into subsets that run on branches of typically some type network tree – but not in running on the entire system.

You’re right – it’s multiple components. Software is just as critical as the network interface card or the router – those are all important – in fact all of those are going to be important for exascale.

TER: And so do you see interconnect as always being a proprietary part of the solution?

Bolding: Not necessarily. I think what is critical when you build for scale is that you are going to end up putting something into the network that the general market doesn’t always need, and I think that’s where you differ from what I would call purely commodity. Pure commodity is driving towards different features in the network than does a network that needs to scale. So Cray does proprietary networks – we do our own interconnect and the software on top of it because we don’t see any commodity infrastructure that would scale. So we believe that is necessary and we want to build systems to scale.

TER: Our perception is that data movement requires more energy than computation. How do you see interconnect technology being applied to address data locality?

Bolding: Well, there are a couple ways. One of the ways Cray has, in the past, dealt with this, is that we build a single network for both data – and for messaging. So there are networks out there that tend to build two parallel networks – one for data I/O and one for messaging. That’s really an energy inefficient way of building a network. What you want is a very fast network that can handle all of the messaging that is available – otherwise you are supporting two sets of wires – two sets of communication. So Cray, with its C-Star and now Gemini – and our future Aries interconnect support a philosophy of sharing the network between both data I/O and messaging.

The second thing is – that it’s true you need to bring components of the network closer and closer together, so what Cray has done in the past is that we have combined the NIC and the router on to a single ASIC. And that minimizes the cost of moving messages on a Cray network. And we’re driving forward for at least another generation of doing that. Now, in more commoditized systems, the NIC is typically on the mother board and then you communicate through the motherboard to the processor chip. We have seen indications that over the course of the next decade that there may be a trend to move some of those network interfaces on to the processors themselves to save energy. And if that’s the case, I think that’s another way in which you can save energy. Cray’s looking very closely at that trend. We’ll work with that trend if that’s what happens, but I think overall you’re correct – you need to bring all these components closer and closer together.

TER: So what have you seen recently in terms of advancements or developments that look rather promising in the area of interconnect?

Bolding: I think certainly what I just talked about in terms of trying to put the data and the messaging on the same network has been one efficiency, and another has been hardware support for special features for scalability and that includes global address space support, it includes hardware support for very fast single side messaging, it includes hardware support for global collective – now some of these are things Cray has and some are technologies our competitors have but I think Cray is driving to include as many of these in our infrastructure as possible. I think all of these are important for exascale and also, sort of what I would call ‘low hop’ topology – topologies that minimize the number of hops that messages have to make on a very large exascale system is going to be very important. I think all of those are technologies that are going to be critical. And optical technologies are going to be critical – especially trying to make those optics and optical technologies affordable. Today optics is fairly expensive. But if we can get the optics to allow for messaging across long distances with low signal loss, then I think that will be a vital technology – but it has to be cost effective in these exascale systems.

TER: And are you guys experimenting with any photonics solutions today?

Bolding: Well, we’re certainly looking at using optics in some of our future interconnects, including the Cascade interconnect. We’re primarily looking at optics on the interconnect. Some are looking at optics within the motherboard or within the ASICS – so there are different ways that different companies are driving this technology.

TER: So, just in general Barry, when folks are talking about exascale systems, what thoughts do you have for those in the community doing research in the area of exascale and where do you think they can push the envelope?

Bolding: Exascale is going to really push the envelope in terms of the scale of the system and the manageability of the system and I think that’s going to be critical to exascale – we really do need to combine the software and the hardware view. It is not just about how many lanes you can get in and out of a processor. If people believe it is all just about injection in and out of a processor, then they are going to fall short. If people think it’s just about being able to boot a very large system and bring It up very quickly – then that by itself is insufficient. If people think it’s going to be just the ability to build big router switches – that’s insufficient. Really, to build exascale, it really is going to take a system view. It’s not going to take an individual component view, and those folks who think that individual technology or individual component view will be sufficient to get productivity out of an exascale system – well, they’re going to be disappointed. They’re going to have a system at the end of the day that may be very good from a single specification point of view — it may have the flops, it may have the injection of one or two other technologies but it will not be productive for the wide community if we don’t take a system view of exascale.

Closing Comments

I would like to say that Cray is aiming for exascale. It’s key to us as a company to really contribute to this exascale initiative. We have produced some of the most productive machines of the last seven or eight years and we’ve driven scale from 10,000 end points to 20,000 end points to 30 or 40,000 end points. When you’re talking about scalable networks with exascale – who knows – we’re talking about 60 and 100,000 end points, and each of those end points are going to be more and more complex and have more and more levels of parallelism built into the end points themselves. And so, if you only look at the end point – and you only optimize the end point – the processor – for exascale, or if you only look at the network and only optimize the network – you really are going to miss it. You have to look at both – and the scale of both is what will drive productivity and being able to use all those endpoints efficiently – and being able to effectively run those programs at high parallelism within a single end point of the exascale system – that’s really going to be the key. And I think I just would like to see Cray contributing the best way we can to the exascale initiative.

For related stories, visit The Exascale Report Archives.