Academic Reviewing

Research is considered a very important part of academic institutions. Ultimately top academic institutions are the ‘temples of knowledge’ and one of the major activities that professors in these temples are involved is not just collecting knowledge from other places but also creating knowledge. This knowledge creation process is a rigorous process.

Most research (knowledge) is initially distributed through what is commonly called research papers. When somebody does research and creates knowledge that he/she considers worthwhile, that knowledge is formally reported in the form of a research paper. Traditionally a research paper is just a draft until it is accepted by experts in the field of research is new knowledge. The process of acceptance as a new knowledge is what is commonly called academic reviewing.

The process of reviewing is usually a very rigorous process. Once a draft is submitted to a journal or a conference in the hope that the community will consider it new knowledge, the editor or a group of editors selects three reviewers (generally) who are supposedly the experts in the field and sends the draft to them for reviews. These reviewers are suppose to comment on many aspects of the draft, ranging from novelty to contributions to language. Editor then looks at the three reviews and decides whether the draft should be accepted or not. if it is not accepted, the comments are used by the researcher (author) to do further research and modification of the draft. This process is fundamental to the progress of knowledge creation. In fact, this is fundamental to the progress of science.

Like most other interesting and useful things in life, there are quirks of this review process also. There are many famous stories about the reviewing that show the problem with this system. For example, the paper written by Tim Barners-Lee (I am reading his book so this is fresh in my mind — there are many similar examples) proposing World Wide Web was rejected by experts by saying that there is nothing new and everything is obvious to people working in the field. Obviously, Tim believed otherwise and continued his research and changed the world by bringing to us the Web (that I am using to post this piece) and became Sir Barners-Lee.

I am talking about this because one of my student just received a review that shows problems with the framework. Here are the reviews :

CONTRIBUTIONS:
This paper uses an event-centric data model for clustering and presenting
news stories. Aa chronology of related stories is produced from the
representation and clusters. The UI to the application permits some degree
of natural language queries as well as a graphical representation of event /
story locations.

KEY POSITIVE POINTS:
The examples from the demo application are fairly interesting and
impressive, showing a nice synthesis of information in the UI. The data
model for the events is also a good, if limited, contribution.

KEY NEGATIVE POINTS:
>From the paper, it is not clear what are the original contributions. Most
of the ingredients have appeared elsewhere or are well-known. The event
data-model is a fine idea, but it’s not clear to me that it stands on its
own as the sole original contribution.

I find this review very interesting because synthesis of knowledge has been the basis of new knowledge all the time. The above reviewer — supposedly an expert in the field — agrees that the work is interesting it accomplishes new things, but the negative point is ‘what is the original contribution’? Most of the ingredients are well-known — so their combination in a unique way to solve a real problem is not considered a new knowledge. Boy, I am sure people will kill to get the formula for CocaCola though all the ingredients used in it are well known.

In fact much of new knowledge is really synthesis of the components and it is this synthesis and the resulting combination that really advances all fields.

Hey, one problem in academic review process is that a reviewer is ‘anonymous’. So no body is suppose to know who wrote the above comments and that allows reviewers to be critical and as human nature is — the anonymity also allows reviewers to be carelessly critical and prejudiced.

Believe me I am not complaining about this particular review because this is one of the minor problems. I am posting this to hear from people about the review process because many communities are concerned about the current review process. Many communities believe that time has come to revamp it significantly. So I took the latest example that I saw and used that here to make a point.

–