A Perspective on Multimedia Computing and Communication (1)

(This is the first post in a series on this topic. I plan to do a book with Gerald Friedland in this area and will share some of my thoughts with you here. Please do provide me feedback — that will help in my thinking and writing.)

Only a few inventions in the history of civilization have impacted society so much in so many different ways and in so many aspects as computers have done. And yet, we have only experienced the tip of the iceberg. The nature of computing is rapidly evolving from simple alphanumeric data to rich multimedia experiences. Computers at one time were used for computing with data; now they are primarily used for information exchange and communication, and are rapidly evolving as a means for gaining insights and experiencing and sharing experiences with our fellow humans across distance and time. In early days of computing, science fiction projected computers in form of robots that will effectively use audio-visual data; later people started dreaming of systems that could organize audio files, images, and video. Now people want to share sensory experiences independent of time and distance. Until just a few years ago, most of the data on computers was alphanumeric; in the next few years most of the data, at least by storage size and bandwidth requirements, will be audio-visual. The fundamental medium of computing and communications was alphanumeric, it is rapidly becoming audio-visual.

Multimedia communication and computing technology has been evolving for quite some time and from different perspectives in different academic and technical disciplines. By its very nature, multimedia requires concepts and techniques from different intellectual disciplines ranging from audio to visual processing; from communication theory to image databases; and from networking to compression techniques. This has resulted in evolution of multimedia computing as a bag of techniques coming from different disciplines. Even worse is that multimedia has become like the elephant in the famous fable about the elephant and the six blind men. Like each blind man, each discipline perceives multimedia only in a very limited aspect. This results in skewed development of the field. We, the humans, use all our senses in unison with our abstract knowledge to form holistic experiences and extract information. The goal of Multimedia computing is to develop communication techniques to allow holistic experiences from multiple sources and modalities of data and extract useful information in the context of various applications.

The fragmented perspective of multimedia has resulted in slow progress in understanding and effective processing of multimedia information though the hardware ranging from sensors to bandwidth has progressed very rapidly. Multimedia computing should utilize correlated and contextual information from all sources to develop holistic and unified perspective and experiences; it should focus on experiences rather than a medium that is only a partial experience.