International Conference on Data Engineering started this morning in Atlanta. The first Keynote talk was by Stuart Feldman from IBM on “The Joys and Challenges of Masses of Moving Data” in which he emphasized the changing nature of data in data engineering (databses as well as search) applications.
A point that he emphasized was that Data in Motion requires rethinking of the approaches that were developed for the Data at Rest. With increasing number of Sensors working all time, it is becoming very important to consider needs of such data in motion. A very important issue is to consider what kind of data needs to be saved and what needs to be discarded. Also when to discard becomes important.
Data engineering is definitely facing interesting challenges because the sources as well as the forms of the data are rapidly changing. How does one convert Data Flood into Information Rivers (using techniques like information extraction from text, audio and video understanding, modeling and analysis of sensory data using different types of meta data) is becoming a real important issue. The process does not stop there, however. These information Rivers must be converted to Knowledge using context of various kinds. Ultimeately, users need Understanding that leads to decisions. In many increasingly complex environments, understanding by humans is achieved using powerful tools for visualization.
It is interesting to see that many people dealing with emerging applications resulting due to progress in technology see that current tools developed for static data are inadequate but legacy pushes many major companies and application developers to continue using the antiquated tools that have lost their relevance. What is required is a major emphasis on tools to deal with dynamic situation. ANd those tools will be more event based than current object-oriented.