Michael Feldman has an outstanding blog post over at HPCwire called “The Secret Life of Supercomputers.” The post talks about some of the less public aspects in the world of supercomputing as divined from a couple of otherwise unrelated news items from the finance and intelligence communities
There were a couple of stories floating around the Intertubes in the past week or so that reminded me of how little we know about large classes of HPC applications. That’s not a good thing.
…What’s the result of all this secrecy? Besides giving the public a skewed view of the industry, it also makes the technology invisible to a larger number of developers. Consider that most HPC apps are still implemented in legacy languages like Fortran and C, while “public” applications for personal computers or the Web are using more modern software frameworks, like Java, .NET, Python, etc. Even though HPC is not a volume industry in terms of software licenses, if more codes were public, you’d probably see a much more rapid development of libraries and tools (which is one reason why CUDA software has developed so quickly). Keeping software in silos makes for a lousy ecosystem.
A very interesting blog post. I agree with the general principle that secrecy can, on occasion, be used for evil as well as for good. And the three examples given were monumentally costly for our country and our grand kids will likely still be feeling the consequences of them.
That said, however, I think the mixing of the notion of public vs private and the different development languages used is confusing and somewhat diminishes that power of the argument. In reality an HPC code can be open or secret no matter the language that it is written in. Most of the open science codes are developed in C and FORTRAN but they are still open.
Furthermore, while I’m no expert on the software development practices of the NSA or the Finance Industry, my guess is that they are direct beneficiaries of the HPC development that is done in the open science community. It produces the pool of people from which they can hire. The research done also produces the methods and techniques they can leverage for their own proprietary purposes, etc.
By definition, true HPC is a leading edge industry. Therefore, there will always be a healthy level of secrecy surrounding it. Leading edge technology is a source of competitive advantage and there will always be a demand for competitive advantage whether it be in business or in government. What is needed is for there to be adequate but not onerous oversight to avoid the kind of major catastrophes that we are now living through.
As our industry matures however, the term HPC is becoming known more and more of an infrastructure technology that is common in a growing number of agencies and businesses. The challenge will be to see the difference between what is leading edge and what is common infrastructure. My guess is that the essence of that difference may lie in large part with the software developed for the more commonly available architecture.