A Dangerous Trade-off : Policymaking in the Era of Big Data

One of the most astonishing aspects of Edward Snowden’s revelations and the consequent congressional hearings regarding the NSA’s data collection activities was the sheer size of the program. While the notion of living in an increasingly connected and digitalized world is widely recognized, it is easy to overlook how this state of constant communication is shaping business and government and what this means for our society as a whole. Viktor Mayer-Schönberger, professor of internet governance and regulation at the Oxford Internet Institute at the University of Oxford, and Kenneth Cukier, Data Editor for The Economist, are certain that we live on the cusp of the next paradigm shift. With this in mind, the authors of Big Data: A Revolution That Will Transform How We Live, Work, and Think set out to build a framework for understanding how technology allows us to process increasingly large amounts of information, how we can use this information, and why big data may be the biggest thing since the printing press.

At the heart of the book is the idea that we are seeing a qualitative shift in knowledge as we increase the size of the information we process. Sampling was a revolutionary instrument, but in many fields it may outlive its usefulness as we now have the opportunity to work with samples so large they come close to including the entire statistical population. The authors make a strong case for building a new methodology for the big data universe. Thanks to several convincing examples, largely from the business sector, Mayer-Schönberger and Cukier argue that, in the future, knowledge must rely less on causal links and more on imprecise data collected from an increasingly large number of sources.

It is counterintuitive that one of the greatest achievements of modern science—precision—is what we must give up in order to achieve more accurate results in the future. Big data, the authors argue, is inherently messy. The crux of their argument holds that approximation on such a massive scale produces reasonably reliable results, and often does so without the need to create models as complex as those required for smaller samples. Examples ranging from grammar correction software to credit analysis indicate that the ability to prioritize quantity over quality can make a big difference. Companies that embrace using data of dubious quality in order to achieve massive scale are more nimble and quick—crucial elements in today’s business world.

Logic may be one of the most powerful tools to be developed by the human mind, yet to use big data we may need to leave logic behind. One of the main consequences of practicing analysis of big data is that we can learn relatively little about causation even as we are able to make predictions more accurately than ever before. Companies like Walmart, Target, and Amazon have successfully found patterns in their consumers’ behavior that enable them to make smarter marketing choices. These patterns do not mean these companies need, or even have the ability, to understand why women purchase unscented lotion during their third month of pregnancy, but it allows them to target the right customers. Mayer-Schönberger and Cukier believe that in a world where correlation can predict the next flu outbreak by tracking Google searches, causal links are no longer as important—and may even be misleading. The idea that causal links are becoming obsolete is one of the more significant and controversial points in the authors’ argument.

While companies may be able to part with causality as they build more effective marketing strategies, it is problematic to encourage this approach among policymakers. If society does not develop the instruments to understand why a child performs poorly in school, there is little to gain from knowing which student will start falling behind first. Moreover, as policymakers rely increasingly on correlation in social matters, it will become harder to avoid profiling...