Reviewing: The Three-Circle Method

Academic peer-review is unquestionably a vast and important topic. On the one hand, it is the key process, which currently can ensure some quality control, particularly regarding fradulent research and plagiarism. Yet, it can also be quite a burden on the academic community as good reviewing takes substantial time and not all review processes and decisions are as transparent/fair as they should be. Hence, many blog entries will be needed to cover the topic. Here I shall propose a simple method to gauge the overall quality of a paper.

Apart from particular details of the work such as certain formulations or presentation style, a key task of a reviewer is to eventually assign a rating. Many journals have a system based upon roughly three categories

(C) reject outright
(B) major revisions
(A) acceptance

Sometimes, there is also a minor revisions category but this tends to mean that upon fixing certain small points, the paper is likely to land in (A). So how to come to a decision? Here I would like to propose a relatively simple first screening test, which I personally call the three-circles approach. As a disclaimer, one should add that this method definitely is far from a full proper review, yet it is far more often than not indicative of the quality of the paper.

threecirclereview

Stage 1 consists in the outmost largest circle as shown in the figure above. This stage involves carefully reading the paper once but at a very slow pace. One should think visually going around the large circle slowly, actually coming back to the promises in the abstract at the end. Via this process, one gets a good indication, whether the paper should belong to category (C). Major flaws tend to be very much visible at first glance, and they tend to come in multiples. Of course, the referee should be critical at any phase during the review. If already multiple major problems are visible with the work at this stage, then it is highly likely that the paper should be rejected. Just describing the objections carefully in the review does often settle the case. If no major objections arise at first reading, my suggestion would be to just leave the paper alone for some extended period of time.

Stage 2 means, we have passed to the middle circle. The smaller circumference represents visually that we should read the paper at a higher pace (after all, the outer circle means we got the broad picture already). This higher pace means going over critical parts multiple times to make sure, what the main idea is and whether it is conveyed in a suitable fashion. The review now really zooms in, it focuses on the core components of the argument. Generally speaking, most papers reveal their true quality at this stage. Either one can find initially hidden faults, deficits, lack of a creative idea, etc, or one cannot. If nothing serious can be found, and the paper is interesting to read, I personally tend to be of the opinion that the paper should probably be published as it almost surely has positive aspects. Of course, the type and level of journal is another matter to be taken into account (in a separate blog!).

Moving over to Stage 3, the smallest innermost circle is limiting the decision to the cases (A) or (B). The smallest circle represents that we have now to traverse it many times to obtain the same distance as on the outer circle. Hence, one can check smaller details, focus attention on potentially missed conclusions or variations, and on judging the precise level of the contribution. Of course, it may happen that even tiny errors propagate to major mistakes. This would then send the paper back to (C). But let us assume that we came to the conclusion that either (A) or (B) are correct. In this case, one can enter in the phase of writing a detailed report. Using the annotated paper with remarks from the three circles is a basis for a good report. How to formulate and phrase such a report is another important matter for a future blog, so stay tuned 🙂

Advertisements

Book Writing

An interesting question of 21st century science is, whether it makes sense to still invest time to write books? Some parts of science have essentially abandoned book writing already. The literature moves almost on a weekly pace in some areas. Maybe even having to look for daily updates in some large-scale laboratory-driven world-wide competitions will be our future. How can it be even reasonable to consider sitting down for multiple years to write a single book? In fact, it could actually be a practical method to deal with some madness.

After all, the scientific endeavor aims to generate long-lasting insight, which is useful not only today, not only tomorrow, but for generations. So there has to be a balance between pushing far beyond the frontier and trying to solidify the frontier itself. Neither extreme approach is going to get us very far. If one just latches out and tries to find hidden gems, the failure rate will be far too high, and one has to probably repeat the process many times to actually have solid knowledge. Just building a solid smooth frontier is also not going to cut it, the innovation level and progress are just too slow. Although it may not look like it, book writing can actually be an excellent way to balance the necessary processes in research.

From my personal experience, I found switching between writing a book (be it a monograph or a textbook) to thinking about research questions a very practical solution. In fact, it can be very frustrating if for several weeks one does not make progress towards a research idea. The thought process narrows down until it is almost impossible to make progress. If one then switches to writing a book, which organizes the results near the current frontier, one is bound to make progress. This yields new motivation and also provides additional insight into current techniques. Then one can use the additional motivation and new techniques to go back to the original research problem. In addition, book writing also indicates, where the gaps in our knowledge frontier are. Thereby, it provides a neat way to find new exciting problems. The interplay between the process of leaping out into the unknown and pushing/polishing the immediate knowledge frontier can lead to a positive feedback loop as shown in the figure below.

bookwriting

One may also view writing a book as a systematic procedure to carry out a scientific version of checks-and-balances. If a field is strictly dominated by hunting for the next large jump, then it can be difficult to systematically reproduce past results. This means a solid foundation is at least partially missing. Stories and concerns about results, which are not reproducible, are certainly available in the scientific literature as shown by just the following examples: click herehere, here, or here.

Never trying to aim high is probably equally as detrimental to a scientific field, progress is bound to asymptotically crawl to a halt. Creative new ideas are discarded as too much out-there by the vast majority in the field. This makes it difficult, if not impossible, to speculate on big ideas. Books tend to generate a broader perspective, they show us much more clearly, what we really know, what we just think we know, and what we may dream about.

In summary, if you ever thought about writing a book yourself, I would recommend giving it a try if certain conditions are met. The main hint is: Planning ahead is key. It will never be finished in the time you calculated initially, and it might be out-dated, or out-competed, if planned incorrectly. Hence, you should probably only try it if you really enjoy it, and if you do not plan to depend upon it in any way. If these conditions are met, I personally think it is a very important experience, and a book can still make major contributions to the general scientific endeavour.

 

The Paper-Writing Process

What is the best way to write a research paper? More precisely, how to communicate the research you do or have done in written form? Obviously there will never be a conclusive answer. Styles differ dramatically depending upon the person. And different approaches can be successful. Nevertheless, there are certainly a few different natural alternatives, all with their own traps and pitfalls. Here I shall try to just give a snapshot of various approaches in the hope that it helps others to reflect upon the practical process of paper preparation.

The main decision one has to make is regarding timing. When do I even have to start trying to write results up? Personally, I believe there are two philosophies:

a) Write-once-finished (let’s call it “wronfi”): The paper is started once the research is finished, i.e., you deem the result you have obtained to be useful and relevant enough for a paper. Then you collect everything you need such as notes, data, simulations, etc and try to compile it into a coherent manuscript.

b) Write-as-you-go (let’s call it: “wraygo”): Every part or result is written up as it is discovered. This may sound like a mad idea. Just immediately trying to pin down everything in a paper format once it is discovered.

In fact, I believe that wronfi and wraygo both have merit, both are called for at times. In principle one may always tweak an experiment or a mathematical generality for an arbitrary period of time. There is always something to improve. Hence, following the wronfi philosophy completely, one never even starts to write a research paper. Writing up any little intermediate result daily is also dangerous. It is possible to get lost in little details. Switching between writing and doing an experiment or calculation can be time-consuming as one has to switch gears almost hourly. Therefore, following the wraygo approach blindly is not going to lead to acceptable results.

Personally, I believe that one should find out, which mix of the two approaches suits your personal style and working hours best. This process should not be taken too lightly. Doing it well usually means increased productivity and this implies additional available hours each day, probably the most valuable resource a scientist can have. But this raises the question: how to actually find out the balance between wronfi and wraygo in your daily scientific ‘diet’? In fact, it essentially does work like trying to lose weight to a certain degree. One can try different strategies and methods but how your body responds is actually quite difficult to determine a-priori. You may be more productive for a while writing things immediately but then struggle to find your focus. This is the well-known jo-jo effect, initially you lose weight just to gain it back again after a few weeks or months.

One trick that worked reasonably well for me is to try to check out those diets, which work out over longer periods of time. For example, when I do have an idea that I would like to write down – e.g. like this blog – then I don’t really feel like waiting. It somehow dilutes my thought process. Unfortunately, this wraygo-type approach has the drawback that I need a suitable computing environment to type everything, otherwise one has to type up hand-written notes, convert things between different formats, or – even worse – keep different versions on different computers (laptop, home, work, etc). However, there is a practical solution by just using a folder in a cloud storage data service. In this folder you can save all your scientific wraygo projects and always be able to add or modify them at minimal opportunity cost. For actual mathematical theorem-proving and doing simulations, I prefer a fixed environment in my office with a computing setup and a corner for doing all the pen-and-paper calculations. This naturally adds a wronfi element to my work. Although this is just one possible ‘diet’, I believe that using a certain scheme that works towards your strengths and tendencies can drastically improve productivity. Of course, there are many other scientists, who have thought about this process. For example, I remember that Terence Tao wrote once on his excellent blog, that he likes to ‘batch-job’ low-brain administrative tasks in one period so he has other periods free for demanding calculations and writing papers. I found that suggestion logical and tried it. Somehow it did not work out directly for me. Initially I had no clue why. The main point was that I had put the administrative batch-job parts on an empty afternoon, did them all in a row, it was horribly boring and I was unmotivated afterwards. The solution was to put them onto a day, where there are already a lot of meetings, i.e., schedule them into the gaps of the day. This has worked out a lot better for me. Maybe for someone else it would work better to schedule them for an evening session, or directly before breakfast. The motivating point I am trying to make is that you should go and try to find out for yourself, it really pays off. Once the correct wronfi-wraygo balance has been determined, I would almost guarantee anyone to be more productive and happy in the paper-writing process.