How to Review a Scientific Paper

Following one of the most popular posts on ConservationBytes.com, as well as in response to several requests, I’ve decided to provide a few pointers for early-career scientists for reviewing manuscripts submitted to peer-reviewed journals.

Apart from publishing your first peer-reviewed paper – whether it’s in Nature or Corey’s Journal of Bullshit – receiving that first request to review a manuscript is one of the best indications that you’ve finally ‘made it’ as a recognised scientist. Finally, someone is acknowledging that you are an expert and that your opinions and critiques are important. You deserve to feel proud when this happens.

Of course, reviewing is the backbone of the scientific process, because it is the main component of science’s pursuit of objectivity (i.e., subjectivity reduction). No other human endeavour can claim likewise.

It is therefore essential to take the reviewing process seriously, even if you do so from only from the entirely selfish perspective that if you do not, no one will seriously review your own work. It is therefore much more than an altruistic effort to advance human knowledge – it is at the very least a survival mechanism. Sooner or later if you get a reputation for providing bad reviews, or refuse to do them, your own publication track record will suffer as a result.

Just like there are probably as many different (successful) ways to write a scientific paper as there are journals, most people develop their own approaches for reviewing their colleagues’ work. But just as it’s my opinion that many journal editors do an awful job of editing, I know that many reviewers do rather a shit job at their assigned tasks. This perspective comes from many years as an author, a reviewer, an editor and a mentor.

So take my advice as you will – hopefully some of it will prove useful when you review manuscripts.

1. Avoid conflicts of interest even before you agree to review the paper. If you have an axe to grind with one or many of the authors listed above the Abstract in the ‘invitation to review’ e-mail, my advice is to decline politely. In the long run, you’ll not help yourself by trying to quash other scientists’ work by going on a reviewing rampage in a futile attempt to prevent publication. You will only piss off the authors, who will eventually guess your identity anyway and attack you mercilessly as a result the next time they get the chance to review your work.
2. If you do accept to review after contemplating the above, read the entire paper. This might seem an obvious and unnecessary point to raise, but I’m convinced that many people who have ‘reviewed’ my own papers haven’t gone past the Abstract. I do not know how many times I’ve had to point out to an editor that the reviewer missed a key component of the manuscript that was (at least to me) rather clearly stated. I’ll admit that in these days of extended supplementary appendices that often exceed the length of the main paper (for the bullshit reason that journal space is ‘limited’), key information can get a little buried in the heap. There are also times where I haven’t stated the information as clearly as I should have, which means a casual read might miss the information in question. Those issues aside, the best way to approach a new manuscript to read it through entirely at least once without focussing too much on bad writing, grammar issues, spelling mistakes or other niggly, minor things.
3. Wait at least half a day before you attempt the full review after reading it once for the first time. This will (a) give you time to reflect on the main message and approach, and (b) allow you to calm down if the authors have happened to claim something that pisses you off (Point 1 notwithstanding). This is the time that you need to pour carefully over the details.
4. Ignore the Abstract (for now). I treat the Abstract (and the title, for that matter) of most papers as advertisements rather than real scientific meat. Even scientists tend to over-emphasise or even sensationalise their findings in the Abstract. It’s best to ignore this until right near the end of your review to increase the chances of providing a (more) objective appraisal.
5. Focus on the big stuff. It’s tempting to start ripping into the authors the first time you come across a turn of phrase that bothers you, or a poorly worded hypothesis. Avoid this. In the interest of efficiency, not wasting your and the authors’ time, and maximising turn-around time, go straight to the heart of the manuscript to determine whether it deserves a more in-depth appraisal. If there is clearly a flawed approach, an obvious and major misinterpretation or worse, a fraudulent and evidence-less claim, then there’s no need to bother picking out the small problems. Do not use this approach as an excuse to provide a one-line review just because you’re a lazy or nasty bastard; rather, use it to avoid spending days to review a paper if it has no hope of passing to the next stage. Again, I remind you of Point 1.
6. Be constructive, not destructive. It helps no one if you’re a prick. It’s easy to say ‘this is shit’ or ‘flawed’ or ‘not worthy of seeing the light of day’, etc. It is much more erudite and helpful to point out ways to improve the problems after you’ve identified them. Nothing discourages authors more from improving their science (after all, isn’t that the point?) than receiving a negative criticism with no offer of a way forward. In other words, be mature and adult about it.
7. It’s perfectly acceptable to disagree. Avoid the logical fallacy that agreement = validation, for we are all human and subjectivity creeps into almost every paper at least to some degree, especially when discussing the uncertainty associated with results or the mechanisms underlying observed phenomena. It’s ok if you disagree with the interpretation – provided the methods appear sound, the results are presented in a clear and objective manner and the hypotheses are clearly outlined, a difference of opinion should not be used as the sole basis for recommended rejection. By all means, state your disagreement, but conceding in such cases can be noble and morally defensible.
8. You are not the editor, so avoid language such as ‘I reject this manuscript …’. That’s not your job nor your right. Stick to the technical aspects of the appraisal and avoid making judgement calls about what the journal should or should not ‘accept’.
9. Always start with the positive. I’ve not always followed this advice, but experience has taught me that a review that launches into the negatives is more demoralising than ones that attempt to point out at least some of the manuscript’s strengths. Don’t be patronising, but I contend that all submitted manuscripts have at least one redeeming feature. Find it and acknowledge it straight up.
10. Minor problems are minor. Being a stickler for good grammar and terminology, I always try to point out where writing can be improved. I think it’s an overlooked element of scientific clarity and that too few authors pay attention to the importance of clear language. That said, put these minor comments at the end of the review and never use them as the sole or main basis for recommended rejection, except in the rare case where the writing is so bad that it completely impedes comprehension.
11. Be timely. This advice is also obvious, but it needs reiterating. Everyone hates delayed reviews – the authors, the editors and the other reviewers. Do not be the lowest common denominator – if you are a slow reviewer, your own papers will be held up longer in the reviewing pipeline as a result.
12. Don’t review too much. There is a tendency among early-career researchers to spend too much time reviewing and not enough time writing their own papers. There’s no hard and fast rule, but I’d recommend reviewing no more than 1-2 papers per month to avoid excessive time lost.
13. Engage with the editor. Remember, (most) editors are human, and so they will often appreciate a discussion via e-mail or telephone if necessary when a manuscript is contentious or borderline. As an editor, I appreciate the frank discussion with expert reviewers in such circumstances when I’m having a difficult time deciding. It’s not unethical to do so.
14. Do not write a thesis. Just like providing a single line is largely useless, a tome of rants and overly prescriptive recommendations is just as unhelpful. It’s not your job to rewrite the paper, even if you are the micromanaging, parental type. Make reasonable recommendations but let the authors do the groundwork for the revisions.
15. Spend no more than 1-2 days per review. If you are spending more than this (full time) on a review, I think you’re wasting your time (see Point 14).
16. Seek advice. While you are morally and legally obliged to ensure author anonymity, in my view it’s entirely acceptable and justifiable to consult another specialists if you are not sure of a particular component of a manuscript. If the technique being presented is entirely foreign to you, or if you know someone who can tell you right away whether or not there is a problem, it’s fine to ask their advice. Do not send them the entire manuscript or reveal the authors’ identity, but by all means discuss the general points with them if you will find it helpful.

There are probably many more sage words of advice on reviewing that I’ve not covered here, so I’m keen to hear of your experiences and other helpful tips.

CJA Bradshaw