In this video from the Perth HPC Conference, Rob Farber from TechEnablement presents: AI is Impacting HPC Everywhere.
The convergence of AI and HPC has created a fertile venue that is ripe for imaginative researchers — versed in AI technology — to make a big impact in a variety of scientific fields. From new hardware to new computational approaches, the true impact of deep- and machine learning on HPC is, in a word, “everywhere”. Just as technology changes in the personal computer market brought about a revolution in the design and implementation of the systems and algorithms used in high performance computing (HPC), so are recent technology changes in machine learning bringing about an AI revolution in the HPC community. Expect new HPC analytic techniques including the use of GANs (Generative Adversarial Networks) in physics-based modeling and simulation, as well as reduced precision math libraries such as NLAFET and HiCMA to revolutionize many fields of research. Other benefits of the convergence of AI and HPC include the physical instantiation of data flow architectures in FPGAs and ASICs, plus the development of powerful data analytic services.
Rob Farber was a pioneer in the field of neural networks while on staff as a scientist in the Theoretical Division at Los Alamos National Laboratory. He is active in the field and works with companies and national laboratories as a consultant, plus teaches about HPC and AI technology worldwide. Rob’s resume includes research positions at NERSC, PNNL, the Santa Fe Institute, and The Center for High-end Computing in Dublin, Ireland. Rob also co-founded two companies that achieved liquidity events (along with a few that didn’t). The first company manufactured and marketed one of the first virtual memory microcomputers and the second combined HPC and machine learning to facilitate the search for drug candidates.
The weighed sum (dot product) is a linear associative memory. If you store only one item the angle between the pattern vector and the weight vector is zero. Store 2 items and the angles between the pattern vectors and the weight vector are likely not zero. To accommodate this the length of the weight vector must increase. This has 2 negative effects:
1/ The system is more sensitive to pattern noise, the variance by the central limit theorem increases with the length of the weight vector.
2/ Since the angles between the patterns and weight vector increase as more items are stored any input pattern that is at a smaller angle than the nearest deliberately stored pattern will cause a large spurious response.
An ReLU neural network is a system of switched linear projections because ReLU is a switch. If an electrical switch is on, 1 volt in gives 1 volt out, n volts in gives n volts out. A 45 degree line when drawn. Off gives zero volts out, a horizontal line. A weighted sum of weighted sums is still a linear system.. Switching those together with ReLU gives a switched system of linear projections. Since the switching occurs at zero there are no sudden discontinuities in the output. You can look up ‘fixed filter bank neural networks’ for more information.
Very likely you need different hardware than is being currently considered.