Brothers Discover Maximal Information Coefficient – A Divining Rod for Big Data?

Print Friendly, PDF & Email

Phillip Ball of Nature reports that brothers David and Yakir Rashef have devised a statistical method that can spot many superimposed correlations between variables. Using this technique, their research team is able to measure exactly how tight each relationship is, on the basis of a quantity that the team calls the maximal information coefficient (MIC).

If you have a data set with 22 million relationships, the 500 relationships in there that you care about are effectively invisible to a human,” says Yakir Reshef. And the relationships are all the harder to tease out if you don’t know what you’re looking for in the first place — if you have no reason to suspect that one thing depends on another.

The MIC is calculated by plotting data on a graph and looking for all ways of dividing up the graph into blocks or grids that capture the largest possible number of data points. MIC can then be deduced from the grids that do the best job.

Read the Full Story.