Big Data

TREND: Vast and expanding data sets acquired by governments and companies through their interactions with Internet users (in conjunction with that generated by scientific research, surveillance and smart object sensors), combined with an accelerated capacity to process and analyse information will expand the possibilities for innovative public/commercial services whilst simultaneously enabling sophisticated profiling of individuals and social groups

The relentless flow of mouse clicks, touch screen interactions, messages, user generated content, credit card transactions, completed online forms and search queries (and more) have all contributed to the generation and acquisition of vast data sets currently held by governments and companies through their interactions with Internet users. According to McKinsey (see page vi) in the US 15 out of 17 industry sectors now hold more data per company than the US Library of Congress (which held 235 terrabytes of data as of April 2011). In addition, the collection of scientific research and surveillance data, coupled with the proliferation of networked devices and smart objects (see the Internet of Things below) has led to a further expansion of these huge stockpiles of data. Rapid improvements in the capacity of technology to process and analyse this data (often in real time) has created new economic opportunities (see page 4). A 2012 report from Intel (see page 4) suggested that innovations in data platforms and analytics enable companies to mine new sources and volumes of information (such as web data and social media data) that were previously too unwieldy or unmanageable to effectively process.

In March 2012 the International Data Corporation forecast that global revenues from the capture and exploitation of big data will reach $16.9 billion by 2015 (IDC 2012). A 2011 report from McKinsey (see page 8) projects the potential value of currently existing data sources could add €250 billion of annual value to Europe’s public sector administration and $300 billion of potential annual value to US healthcare. However, the benefits of increasingly intelligent and automated data collection and processing on such an unprecedented scale must also be balanced against concerns about the privacy and security of personal information. The aggregation of multiple data sources could allow organisations and governments to build sophisticated profiles of individuals without their knowing consent. A consumer backlash against this trend could undermine the future availability and legality of such potentially commercially/socially valuable processes.

Building upon the trend toward big data, the number of smart objects equipped with sensors and the capacity to communicate online is expanding at an exponential rate. According to estimates the number of networked devices exceeded the number of people on the planet for the first time in 2011 (World Economic Forum 2012, page 47). Industry projections suggest that by 2050 the scale of automated machine-to-machine traffic could mean that connected devices outstrip the number of connected human beings by six to one (Ibid). The exploding quantities of sensory and environmental data produced by these devices (ranging from pacemakers and tumble driers to street lights and vending machines) – coupled with increasing capacity to rapidly administer and analyse large data sets – will facilitate the development of complex automated services and smart objects ranging from everyday appliances to infrastructure.