Big Data: The Information Overload Challenge

Big Data: The Information Overload Challenge

Lt Gen AKS Chandele
Lt Gen (Dr) AKS Chandele PVSM, AVSM (Retd)
Managing Editor
[email protected]

World creates 2.5 quintillion bytes of data every day, says IBM, adding that 90 per cent of data present today did not exist two years ago. Google has recently stated that it creates the same data in two days (5 exabytes ) that was created from the dawn of time till 2003.

Three things have facilitated the generation of such huge amounts of data, first being the digitisation of data by adopting the binary system which lends itself to easy storage and manipulation of data. It is estimated that 94 per cent of all data was stored digitally in 2011 as compared to less than 1 per cent two decades ago. The proliferation of a variety of sensors, electronic business/ industrial machines and communication devices enables acquisition of huge amounts of text, audio and video data unobtrusively, almost as a by-product of the functions that these devices perform. Walmart alone handles 1 million customer transactions an hour, resulting in 2.5 petabytes of data. 96 per cent of the world today is connected by mobile communication devices, who with their cameras and GPS are producing humungous amounts of data. Twitter, Facebook and Google add to the mayhem. Finally, the availability of progressively larger and relatively cheaper storage capacities ensures that this data is stored easily, without the need to delete to make way for fresh data.

However, it is not just the volume that lends the name ‘big data,’ but the inability to process these data sets with commonly used software tools. Processing data produces information, further processing it provides valuable knowledge or intelligence. The value of data being acquired lies in its analysis. For the commercial sector and in governance, it provides valuable insights for better decision making and feedback for course correction.

Intelligence derived from big data is critical for defence and security agencies and cannot be neglected at any cost. With the deployment of increasing numbers of satellites, UAVs and a variety of battlefield sensors, the information gathering capabilities of militaries have increased manifold. What is woefully lacking is the corresponding increase in analysis capabilities. Though automation based on customised software is helpful but the need for analysis by experts is indispensable.

Intelligence requirements for defence are at three levels – strategic, operational and tactical. While some time may be available so that big data can be analysed to provide useful intelligence for strategic and operational planning purposes, data analysis to derive intelligence for tactical operations needs to be real-time, or in as short a time as possible, in the range of a few seconds, to be of any value. The information management challenge lies not so much in acquiring the data but in separating the wheat from the chaff, and not getting drowned in data.