This week they announced a new version of Parabon Crush with special sauce developed through their relationship with NASA
…Now enhanced with a search algorithm developed for NASA, Crush uses the idle capacity of thousands of computers on a Frontier Grid to discover optimal statistical models within high-dimensional datasets that are too computationally challenging for traditional approaches.
Crush statistical models can be used to identify hidden correlations, explain variability and predict future phenomena in practically any domain. Dr. William Petros, Pharm. D., of the Mary Babb Randolph Cancer Center, has used Crush to identify genomic markers that indicate whether a patient will respond to particular chemotherapies. He stated, “Crush revealed significant explanatory models that previous conventional approaches used by experienced, Ph.D. statisticians failed to find.” In industry, Crush has been put to such diverse uses as forecasting stock performance and generating psychometric models of consumer preferences. Government customers have used it to forecast contract overruns and determine the causal factors for the spread of West Nile Virus.
The technology in the old version was, according to the release, um…slower
For example, it would take 4M years to exhaust a dataset with just 100 factors, even using 10B computers, each evaluating 1M models per second. The latest version of Crush employs a novel evolutionary algorithm that searches arbitrarily large model spaces deeply and effectively without needing to exhaust them. The so-called Opportunistic Evolution (OE) algorithm, which is part of Parabon’s Origin Evolutionary Software Development Kit, is a combination of genetic algorithms specifically designed to maximally leverage grid-scale capacity, enabling Crush to discover important statistical relationships that are practically impossible to find otherwise.
Crush can run as an Excel plugin, or from the command line.