A Guide for AI Reviewers: What Makes a Paper Different?

One of the annual top conferences in the field of artificial intelligence, CVPR 2022 (Conference on Computer Vision and Pattern Recognition), has been released, and rebuttal has also closed submissions. With the sharp increase in the number of submissions to the top papers, there are not enough people to review the manuscripts, and the quality of the review has also been criticized a lot.

Zhihu netizens sorted out the statistics of the number of submissions and the number of middle drafts in recent years

CVPR uses multi-layer double-blind peer review, and in recent years, the top review has been announced, and it is known that the spit on the reviewer's comments has become an annual must-have movement. Recently, the computer vision field big man Michael J. Black also blogged to join the discussion. "What makes a paper different? One of the focuses of many reviewers is novelty. But what is the novelty of science? Blake asks questions at the beginning.

Blake is professor emeritus of the University of Tübingen and one of the founding directors of the Max Planck Institute for Intelligent Systems in Tübingen, Germany, where he leads the Perception Systems Division. Blake is a Member of the National Academy of Sciences of Germany, a Member of the National Academy of Sciences, and a Foreign Fellow of the Royal Swedish Academy of Sciences, and has received the IEEE Computer Society Distinguished Paper Award (1991), the Marr Award Honorable Mention (1999 and 2005), the Koenderink Award in 2010, the Helmholtz Prize in 2013, and the Longuet-Higgins Award in 2020.

Blake observes that reviewers often mistake complexity, difficulty, and technicality for novelty. In peer review in the scientific community, novelty seems to hint at these things. It might be better if we removed the word "novel" from the review notes and replaced it with beauty.

Blake argues that beauty eliminates the concepts of "technology" and "complexity" and moves closer to the core of scientific novelty. A painting can be beautiful even if it is simple and technically complex. Thesis is also available. A few of Picasso's paintings can be as beautiful as one of Rembrandt's intricate paintings.

"Keeping aesthetics in mind, let's take a look at some common reviewer misconceptions about novelty." Blake wrote.

The original blog post compiled by the following reporters:

Think of complexity as novelty

The simplicity of an idea is often confused with the lack of novelty, and the opposite is true. A common review criticism is "It just changes one term in the loss and everything else is the same as prior work.")

But if no one has thought about making such a change, then it is actually novel. Creative insight is to recognize that a small change can have a big impact and formulate the new loss.

This kind of review makes my students say that we should make an idea seem more complex so that the reviewer will find it more valuable. I value simplicity over unnecessarily complex; the simpler the better. Better science is about taking an existing network and replacing something with one, not concocting a whole new one but just to make it look more complex.

Think of difficulty as novelty

It's hard for a paper to get into top conferences, so reviewers often feel that ideas and technical details have to be hard, and that authors must "bleed, sweat, and cry" to be worthy of a good paper. Inexperienced reviewers, in particular, prefer to see the author really working hard.

Crafting a simple idea means removing unnecessary things to reveal the core of things, which is one of the most useful things a scientist can do.

A simple idea may be important, but it can also be trivial. This is where reviewers struggle: a small idea is an unimportant one. If a paper has a simple idea that is better than existing technology, then it is likely not trivial. The author has grasped some of the cores that people in the field will be interested in.

Think of surprises as novelty

Novelty and surprise are closely related. By definition, a novel idea is a surprising idea – something no one in the field has thought of. But there's another side to this, because surprise is a fleeting emotion. If you hear a good idea, there will be a moment of surprise, and then, the better it is, the more obvious it will look. A common comment is that the idea is obvious, and the author simply combines two well-known ideas.

Obvious is the opposite of novelty. So, if an idea takes it for granted after you've heard it, reviewers quickly assume that it's not actually new. However, novelty must be evaluated before this idea emerges. If it's easy to explain and obvious in hindsight, then this in no way diminishes the creativity and novelty of the idea.

Treat technological novelty as novelty

The most common misconception among reviewers is that novelty is related to technical details. Novelty (and value) appears in many forms in papers. If a new dataset does something that other datasets haven't done, then it may be novel, even though all the methods used to generate the dataset are well known. If no one has thought of using the old method in this way, then the new use of the old method may be novel. Replacing complex algorithms with simple ones can provide insight.

Novelty reveals itself in as much of a way as beauty. Before criticizing a paper for lacking technical novelty, ask yourself if the real novelty is elsewhere.

Think of usefulness or value as novelty

Not all novel ideas are useful, just new properties don't mean value. What we want is new ideas that lead us to a certain goal. Here, reviewers need to be very careful because it's hard for you to know where a new idea is going in that space, because any predictions we make are based on the current field.

A common comment I've seen is that the author describes a new approach, but I don't know why anyone needs this.

The lack of practicality is indeed a problem, but it is difficult to evaluate with a new idea. Reviewers should be careful here and realize that the imagination of all of us is limited.

Personal experience sharing

My early career was built on observing and formalizing the connections between two existing fields: robust statistics and Markov random fields. The novelty stems from the fact that no one has put these ideas together before. It turned out to be a fertile space with many surprising connections and new theories. Fortunately, these connections have also proven valuable, resulting in state-of-the-art practical algorithms.

In hindsight, the link between robustness detection and computer vision science seems obvious. Today, the use of robust detectors in computer vision has become the norm and doesn't seem to be any newer than breathing. But seeing these connections for the first time before someone else sees them is like taking a breath for the first time.

When you catch a glimpse of a new way of observation, there is nothing more exciting in life than a spark that appears in a flash in science. You will feel as if you are the first person to stand on a mountain peak, and you see the world in a way that you have never seen before. It's novel, it happens in an instant, but is facilitated by all of one's experiences.

The resulting paper embodies the process of translating ideas into code, experiments, and text. In this translation, the beauty of the spark may only be glimpsed, and my request to the reviewer is to try to imagine the darkness before the spark.