We are witnessing the ‘rise of Big Data’ in the last few years. The growth in the rate of the production of the data seems to be only accelerating. So one needs to ask a very important question – what is the real source in the creation of the big data and how can it be harnessed efficiently.
A major source of big data is the dynamic world. New events are happening at all levels and these events result in new data. These events were happening earlier also but the technology for capturing experiential data related to events did not exist and the storage costs were too high for people to consider storing it. Now that we have all kinds of sensors, including so called human sensors, to capture experiential data and the cost to store and distribute this are relatively low, big data has arrived. And the number of sensors is likely to keep increasing at rapid rate so the amount of data being created will also increase. Similarly, number of people using mobile phones and creating information, as a human agent and human sensor, is raqpidly increasing. It is expected that currently in early 2012 the number of mobile Internet user is around 1.5 Billion and it will rise to around 5 Billion in 2020. That itself will start contributing significantly to Big Data.
Much of the Big Data is somehow related to events. Even the knowledge that is extracted from the big data is mostly derived through the analysis of events represented in big data. In an interesting geeky sense, events are analyzed to extract and communicate ‘stories’ related to a specific theme. These themes could be the growth of economy in the last 5 years, or improvement in quality of life in Asian countries in the last 20 years, or causes of global warming, or growth of urbanization in China and India, or any other similar thing. The most important thing is that all such abalysis is based on the event analytics in Big Data.
In a very practical sense, Big Data is used to tell Mega Stories. Mega stories, at the other extreme from the micro stories discussed earlier, are used to tell a story that could only be created by considering a large volume of relevant events in the big data. All these events must be selected and aggregated based on the goal of the storyteller.