Peer Review: Solemn Duty Or Merely Extra Work?

The other day on Twitter I saw Rolf Zwaan tweeting about Collabra's policy of rewarding reviewers with credit that can be traded in for credit for organisational article processing fees (APC) budgets, into Collabra's fee waiver account, or for cash to the reviewer. This idea, of paying reviewers for reviewing, comes up now and again on Twitter and it always sounds wrong to me. My gut response is that we should not be paying individuals money to review articles.

So I did the normal thing and posted a Twitter poll, with the following results:

Thesis: getting paid for peer review is a bad idea because it's part of our collective duty to improve science— Andrew & Sabrina (@PsychScientists) April 13, 2016

Turns out I am in the minority! The various discussion I've had around this tweet have been kind of interesting too. 
People who think getting paid for reviewing fall into roughly three groups

1. People who don't expect it but aren't opposed, because it doesn't seem weird or inappropriate
2. People who think reviewing is work, and like all work reviewing should be recognised
3. People who are pissed off at publishing profits and think we should get a slice, given that our peer review is part of the labor that creates those profits

I just disagree with the first people, which is fine. 

The second group: I don't object to me getting money in exchange for work, and neither does my mortgage lender. My feeling is that I already get paid to science, and part of sciencing is peer reviewing. More on that below.

The third group: I get this feeling, I really do. But I believe publisher profits and payment for peer review are two separable issues, and while it's true that out in the world these issues are tangled, I'm mostly interested right now in thinking about payment for peer review.

My theory of what peer review is
The Collabra model is actually useful, because it helped me identify what specifically was weirding me out. I'm not opposed to reviewers earning recognition. Earning credit which I could trade in to pay forward for APC credit seems excellent to me, and in exactly the right spirit. Some of the second group also talked about getting peer reviewing explicitly into our workload/deployment models so that our workplaces understand what we are doing, and I'm on board with the spirit of this too.

Other activities also come with extra money; Chris Chambers & James Kilner mentioned honoraria for editing, lecturing, PhD vivas etc and wondered if I was opposed to these too. Editing and lecturing strike me as genuinely extra work, so extra payment feels appropriate. PhD vivas are an interesting gray area for me; it shares lots of features with reviewing but also with things that are extra work.

Most of these seem fine to me. It turns out that what bugs me is earning extra money for reviewing, as if it is something we would not ordinarily do but are doing on top of our normal job. I think what bugs me is that I firmly believe peer reviewing is an intrinsic part of my job already.

My theory (which is mine, etc :) of peer review is that it is a duty. It is a responsibility. It is a service that we owe the scientific community. Peer review in practice is flawed, but the principle of peer review is that I, the reviewer, get to read and evaluate something a fellow scientist is offering the field and I am making sure that it is of sufficient quality to be worth entering into our collective knowledge. My solemn duty is to ensure the methods and analyses are up to the challenge of answering the question being posed (I peer review in the PLOS One style and always have - it's not my job to judge the potential impact, etc, these things happen later). A new article should add as much signal and as little noise to the literature as possible and I'm part of the process to ensure that. 

I believe all this is part of my role as a member in good standing of the scientific community, and I think that getting paid extra, as if this work was something above and beyond what I'm already paid for, is just a category error.

I may be wrong, of course, and my poll certainly suggests that while I am not alone, I am not in the majority. Honestly, I feel a little sad about that, and I worry that if early career researcher think peer review is somehow extra work we haven't been teaching them properly. 

Summary
I fucking love reviewing papers. I am honored every time I do it that someone considers my opinion worth asking before publishing a paper. I work hard on reviews, I try my hardest to be constructive, and I sign them because that helps keep me focused on being constructive (and because authors deserve to know who reviewed their papers).

Are we getting screwed over by giving this labor to publishers for free? Sure, and that's something we should worry about. Isn't post-publication peer review the future of reviewing? Maybe, and I like that we're doing it more, but I still think there's a place for pre-publication checks. Isn't peer review wildly flawed, full of biases and nonsense? Hell yes, but I'm deeply offended by this nonsense* and I believe we should all do better. But at the end of the day, I think we should treat peer review as the solemn duty that it is, and we should work our asses off to make the process achieve it's aim of quality control. 

(*The all time worst I saw was for a paper I reviewed that was a detailed and technical perceptual modelling paper. I sweated blood over the details, figured out what I like and what I thought needed work, and generally took the whole process very seriously. When I got cc'd in on the editor's decision letter, I saw Reviewer 2's whole review said

I'm not an expert in this kind of modelling but it seems fine to me. Accept.

I don't know what offended me more; the fact that a reviewer felt that this review was appropriate, or that the editor did (his letter made it clear our reviews had equal weighting in the revise-and-resubmit decision). This was 10 years ago, and it still burns.)