Too Many Reviewers

(reposting here to archive. Discussion is like to be on scatter.wordpress.com)

I freaked out recently when, after reviewing an article, I received a packet of FIVE (5!!!)  reviews on the same article. I chewed out the editors for wasting my time and told them I would never review for their journal again. After an exchange (in which I got a little less testy), I told them I’d post my concerns to scatterplot and open a discussion on the topic. Although five was over the top and freaked me out, it has become pretty common now for me as a reviewer to get a packet with four reviews. No wonder we regular reviewers are feeling under the gun. The old calculation of two or even three reviews per article has gone by the wayside. The pressure for fast turnaround and the high turn-down or non-response rate among potential reviewers has led editors to send out articles to extra reviewers in the hopes of ending up with at least the minimum two or three.

But this is a death spiral. As a frequently-sought reviewer I get at least four requests a month, sometimes as many as eight, and I used to get more before I got so crabby.  When I was young and eager, I was reviewing an article a week [and thus, by the way, having a huge influence on my specialty area], and I know some people who are keeping that pace. But at some point you burn out and say “no more.” I, like all other frequently-sought reviewers I know, turn down outright the requests from journals I don’t know for articles that sound boring, and then save up the other requests and once a month pick which articles I want to review. So the interesting-sounding articles from good journals get too many reviewers, while the boring-sounding articles from no-name journals get none. If journal editors respond to the non-response by reviewers to boring-sounding articles by sending out even more reviewer requests per article, our mailboxes will be flooded even more and the non-response rate and delayed-response rate by reviewers will go up even more. Senior scholars are asked to review six to eight (or more?) articles per month. You have to say no to most of the requests.

And then we have the totally out of hand R&R problem. I think it is completely immoral to send an R&R to ANY new reviewers. I know a young scholar with a perfectly good paper who is now on the 4th (!!!!) iteration of an R&R from ASR. Not because she has not satisfied the original reviewers, but because the editors keep sending each revision to a new set of reviewers in addition to the original reviewers and, of course, the new reviewers have a different perspective and a new set of suggestions for the paper, some of which cover ground that was gone over in one or more of the previous revisions. Not to mention the problem that R&R memos are now longer than the original articles!!  We are no longer a discipline of article publishing, we are turning into a discipline of R&R memo-writing.

Something has to change.  Senior scholars burn out and get reputations for being difficult, possibly because editors don’t know how many other people are asking them to do things. Junior scholars would want to review and wonder why nobody is asking them, and other junior scholars think they are being tapped a lot because they are getting four requests a year. Article-submitters (disproportionately junior scholars) whine and complain about slow turn-arounds, and imagine — what?  I guess I don’t know what they imagine? Do they even understand what is happening on the reviewer side of the equation? I think some of the more clueless imagine that reviewers are just queuing up to write negative reviews about them and it is all the editors’ fault for not organizing things better.

My purpose in posting is to open the discussion. I think what is needed are some ground rules that would help the senior scholar problem. (1) Reviewer time is a scarce resource. Treat it as such. Do not waste people’s time. (2) No article is ever sent to more than three reviewers. Better is to send to two and ask for a third if there is a split vote.  (3) If a reviewer fails to respond in a timely fashion, they get an email: please respond or we will send the article to someone else. (4) If an editor has three reviews, they immediately send a notice to anyone else they asked for a review saying “we have enough now” or, if you insist, “we have three reviews but they are mixed, and your opinion would help.” (5) If you get two reviews and the situation is obvious, tell anyone else you asked for a review “never mind.”  (6) An R&R is sent back to the original reviewers and to NOBODY ELSE unless there is some very specific issue and the paper author is told at the time what the issue is and the category of additional reviewer who will be solicited. (7) Author angst about turn-around time is dealt with not by sending articles out to eight possible reviewers (!!!!) but by keeping authors informed of their status. Telling an author that they are having a hard time getting reviewers lets them know what is going on. (8) Tell reviewers you want a response to the “will you review?” email within two weeks and cancel the invitation if they do not respond within one week to the follow up to the initial request. Leaving the requests open just encourages the kind of gaming I described and increases the risk of wasting reviewers’ time with too many reviews.

To expand the pool of reviewers among junior scholars, it seems to me that there needs to be a database set up of potential reviewers. This would have to have cvs and samples of the person’s own publications/writing. Does anyone have an idea about how to get such a thing going?

Scatterplotters: Your thoughts?

blog commenting, blinding articles

I’m in California helping my mother, who is much better than she was in March, but still mostly housebound, on oxygen, and weak. She’s easily frustrated and demanding, which is both understandable and tiring. One upshot is that I’ve been blog commenting all day. I do get periods of free time, but as I never know when I’m going to be interrupted, it is hard to crawl into the writing I’m way-overdue on. So far I’m 0 for 0 in my goal of working at least two hours a day, although I have gotten some reviews done.

Apropos of bits of work, here’s a question about articles for review. Say there is a pretty well-known data set that is identified with exactly one research team, such that anybody who has read the literature will recognize the authors, or at least the PI, from the data. It does not matter whether you use “identifying reference withheld” or third person references, the author, or at least the author’s research team, will be identified. How much trouble should the author go to in attempting to present the appearance of conforming to the norm of not identifying the author, i.e. in using the third person in describing the research procedures? (Identifying reference withheld would seem absurd in this situation, especially as the withheld citations would be central to any literature review.)

My view as a reviewer: if the author of an article in my area is a senior person, I know who it is. But under the cloak of the anonymous reviewer, I am willing to say critical things about my friends. And good things about people whose work I don’t recognize (and thus know must be junior). I have been told by editors that this is common; people will not say bad work is good just because their friends wrote it. It is the anonymity of the reviewer that is central to the integrity of the process. So I don’t think trying to pretend anonymity is worth the trouble. In fact, I loathe “identifying reference withheld”!! If you are going to anonymize, do it with third person references.

What do you think about an author who does not even go through the motions of third person references to disguise authorship, in a case where disguise would be futile anyway. Is that bad form? Or understandable, let it go? Should editors police this kind of thing? Do they?