Big Data Hasn't Changed Everything

Big Data Hasn't Changed Everything: Technology has a long way to go in mapping the variables of human life - Philip Delves Broughton, Wall Street Journal

The more I hear the term "big data," the more suspicious I become. Not in an Edward Snowden, the evil government's spying on us sort of way. If the curious of Fort Meade, Md., the National Security Agency's home, wish to poke through my electronic sock drawers for signs of terror, they are more than welcome. Happy to do my bit for national security.

No, the problem comes when the term becomes ubiquitous. It's one thing for the NSA's quants or scientists at the Large Hadron Collider or genome sequencers to talk about big data. Big nails need big hammers, and the phenomenon of big data is certainly real. The three Vs of volume, velocity and variety, coined by the techies at the Gartner Group, IT +1.25% have created a gusher of data that clever minds can use to great effect.

But it's quite another thing when you start to hear how big data is going to upend everything. In their essential new guide to the subject, "Big Data: A Revolution That Will Transform How We Live, Work and Think," Viktor Mayer-Schönberger and Kenneth Neil Cukier write: "Society will need to shed some of its obsession for causality in exchange for simple correlations: not knowing why but only what. This overturns centuries of established practices and challenges our most basic understanding of how to make decisions and comprehend reality." We need to learn to trust what the data is telling us before we fully understand why.

They add that such a drastic change to how we weigh evidence and decide will take a lot of fine tuning. Ethics, morality, civil liberties, everything risks being thrown under the big-data bus, unless we are exceedingly careful.

Fortunately we have one giant data set around which to pivot this discussion. In the 1980s, the financial industry was transformed by the arrival of what at the time seemed like very big data indeed. Brokers were removed from trading floors and replaced by digital exchanges. Money which once bumped into borders now started to flow unimpeded. Myriad frictions were removed and the industry boomed. Computer scientists started popping up inside banks and hedge funds to identify trading opportunities within the great torrent of data. Many were and are truly brilliant. In a war, you would want them on your side breaking enemy codes.

Things went awry when everyone started to think they could do it. When the decidedly adult pursuits facilitated by the new torrent of data fell into the hands of intellectual children, who had no real idea of what they were doing. When the likes of Citibank and Societe Generale GLE.FR -1.35% started to think they could play with the best of the hedge funds was the time to start stuffing your cash in a mattress. When all you do is watch data models for instructions, for whats, with no idea of the whys, you become easy prey for the monsters of risk.

What seems blindingly obvious in retrospect, that there was no alchemy capable of turning subprime loans into sure-thing derivatives, was either missed or intentionally glossed over by Wall Street's big-data thinking.

Managers are constantly being told that they are one hardware or software installation away from business nirvana. But they should bear in mind the lessons of the financial crisis the next time a consultant waltzes into their office declaring big data the next big thing, or even worse, a "paradigm shift."

Because big data as a technological opportunity and big data as a management theory are two separate things. However much big data can yield, information will never be perfect. As efficient as these data models become, managers will still have to make decisions with limited certainty about the outcomes. Data helps and has since the scouts of ancient armies returned with reliable numbers. Eisenhower at D-Day had more data than Hannibal at Cannae, but waging war remained a beast of a task. The challenge for managers has always been the human mind and heart, which seems punier than ever in the shadow of the terabyte.

Consultants are already whipping up a flurry of winking big-data dashboards for managers, with every organizational activity reduced to a few key numbers. But still there will be rogue traders, rats in restaurant kitchens, and drunk machine operators. If big data is to escape the graveyard of managerial fads—knowledge management anyone?—it has a lot to prove.

Data has been big for a while now and is getting exponentially bigger. But we shouldn't feel inadequate because we rely on our animal traits like gut, intuition and bias. Technology has a long way to go in mapping the variables of human life. And the moment it starts to feel like tyranny, we have one lethal weapon in our arsenal. It is called the off switch.

Mr. Broughton is the author of "The Art of the Sale: Learning from the Masters about the Business of Life" (Penguin Press, 2012).