Academic peer-review is unquestionably a vast and important topic. On the one hand, it is the key process, which currently can ensure some quality control, particularly regarding fradulent research and plagiarism. Yet, it can also be quite a burden on the academic community as good reviewing takes substantial time and not all review processes and decisions are as transparent/fair as they should be. Hence, many blog entries will be needed to cover the topic. Here I shall propose a simple method to gauge the overall quality of a paper.
Apart from particular details of the work such as certain formulations or presentation style, a key task of a reviewer is to eventually assign a rating. Many journals have a system based upon roughly three categories
(C) reject outright
(B) major revisions
Sometimes, there is also a minor revisions category but this tends to mean that upon fixing certain small points, the paper is likely to land in (A). So how to come to a decision? Here I would like to propose a relatively simple first screening test, which I personally call the three-circles approach. As a disclaimer, one should add that this method definitely is far from a full proper review, yet it is far more often than not indicative of the quality of the paper.
Stage 1 consists in the outmost largest circle as shown in the figure above. This stage involves carefully reading the paper once but at a very slow pace. One should think visually going around the large circle slowly, actually coming back to the promises in the abstract at the end. Via this process, one gets a good indication, whether the paper should belong to category (C). Major flaws tend to be very much visible at first glance, and they tend to come in multiples. Of course, the referee should be critical at any phase during the review. If already multiple major problems are visible with the work at this stage, then it is highly likely that the paper should be rejected. Just describing the objections carefully in the review does often settle the case. If no major objections arise at first reading, my suggestion would be to just leave the paper alone for some extended period of time.
Stage 2 means, we have passed to the middle circle. The smaller circumference represents visually that we should read the paper at a higher pace (after all, the outer circle means we got the broad picture already). This higher pace means going over critical parts multiple times to make sure, what the main idea is and whether it is conveyed in a suitable fashion. The review now really zooms in, it focuses on the core components of the argument. Generally speaking, most papers reveal their true quality at this stage. Either one can find initially hidden faults, deficits, lack of a creative idea, etc, or one cannot. If nothing serious can be found, and the paper is interesting to read, I personally tend to be of the opinion that the paper should probably be published as it almost surely has positive aspects. Of course, the type and level of journal is another matter to be taken into account (in a separate blog!).
Moving over to Stage 3, the smallest innermost circle is limiting the decision to the cases (A) or (B). The smallest circle represents that we have now to traverse it many times to obtain the same distance as on the outer circle. Hence, one can check smaller details, focus attention on potentially missed conclusions or variations, and on judging the precise level of the contribution. Of course, it may happen that even tiny errors propagate to major mistakes. This would then send the paper back to (C). But let us assume that we came to the conclusion that either (A) or (B) are correct. In this case, one can enter in the phase of writing a detailed report. Using the annotated paper with remarks from the three circles is a basis for a good report. How to formulate and phrase such a report is another important matter for a future blog, so stay tuned 🙂