Big Money for Companies That Can Analyze Big Data

As data volumes skyrocket, startups looking to take advantage of the opportunity in Big Data need to focus on the art of statistical-learning algorithms, says Michael Driscoll, founder of Dataspora and co-founder of Metamarkets (see full disclosure below.)

Driscoll took the stage of the IA Ventures Big Data Conference in New York today and laid out three skills necessary for data-driven start-ups: data munging, the corralling and wrestling of data; modeling, the statistical analysis of data through algorithms; and visualization, the presentation of all the data. While all three are necessary for success, Driscoll believes that modeling and analysis through algorithms is what will determine winners and losers in Big Data.

In the past, analysis might have meant just organizing the data into counts, sums and averages. But new data-product companies will have to take analysis to the next step toward powering recommendations and predictions, the kind of work being done by Pandora, Amazon or Netflix to suggest content to users

“The secret sauce is predictive analysis powered by data,” said Driscoll. “It’s less about what you did and more about what you should do, and not even telling you what you should do … it should just do it for you.”

This means that the two other skills — munging and visualization — must recede into the background to some extent. According to Driscoll, data-driven companies are spending some 80 percent of their time data munging, basically ingesting data and preparing it for analysis. He said that’s due to the hard work involved in wrestling with various data sets and managing the friction as data comes in, something he said should become easier over time. There’s also a lot of time spent on visualization, which allows customers to see the data and make decisions from it.

For companies looking to get ahead, he said the key lies in investing in machine-learning algorithms. He pointed to the demise of Swivel, a data visualization company, as a sign that visualization isn’t enough. Swivel’s executives said recently that the company had trouble articulating the usefulness of visualization and prioritizing new features.

“Visualization gets them in the door,” Driscoll said, “but what matters is search results or recommendations.”