Too Many Reviewers

(reposting here to archive. Discussion is like to be on scatter.wordpress.com)

I freaked out recently when, after reviewing an article, I received a packet of FIVE (5!!!)  reviews on the same article. I chewed out the editors for wasting my time and told them I would never review for their journal again. After an exchange (in which I got a little less testy), I told them I’d post my concerns to scatterplot and open a discussion on the topic. Although five was over the top and freaked me out, it has become pretty common now for me as a reviewer to get a packet with four reviews. No wonder we regular reviewers are feeling under the gun. The old calculation of two or even three reviews per article has gone by the wayside. The pressure for fast turnaround and the high turn-down or non-response rate among potential reviewers has led editors to send out articles to extra reviewers in the hopes of ending up with at least the minimum two or three.

But this is a death spiral. As a frequently-sought reviewer I get at least four requests a month, sometimes as many as eight, and I used to get more before I got so crabby.  When I was young and eager, I was reviewing an article a week [and thus, by the way, having a huge influence on my specialty area], and I know some people who are keeping that pace. But at some point you burn out and say “no more.” I, like all other frequently-sought reviewers I know, turn down outright the requests from journals I don’t know for articles that sound boring, and then save up the other requests and once a month pick which articles I want to review. So the interesting-sounding articles from good journals get too many reviewers, while the boring-sounding articles from no-name journals get none. If journal editors respond to the non-response by reviewers to boring-sounding articles by sending out even more reviewer requests per article, our mailboxes will be flooded even more and the non-response rate and delayed-response rate by reviewers will go up even more. Senior scholars are asked to review six to eight (or more?) articles per month. You have to say no to most of the requests.

And then we have the totally out of hand R&R problem. I think it is completely immoral to send an R&R to ANY new reviewers. I know a young scholar with a perfectly good paper who is now on the 4th (!!!!) iteration of an R&R from ASR. Not because she has not satisfied the original reviewers, but because the editors keep sending each revision to a new set of reviewers in addition to the original reviewers and, of course, the new reviewers have a different perspective and a new set of suggestions for the paper, some of which cover ground that was gone over in one or more of the previous revisions. Not to mention the problem that R&R memos are now longer than the original articles!!  We are no longer a discipline of article publishing, we are turning into a discipline of R&R memo-writing.

Something has to change.  Senior scholars burn out and get reputations for being difficult, possibly because editors don’t know how many other people are asking them to do things. Junior scholars would want to review and wonder why nobody is asking them, and other junior scholars think they are being tapped a lot because they are getting four requests a year. Article-submitters (disproportionately junior scholars) whine and complain about slow turn-arounds, and imagine — what?  I guess I don’t know what they imagine? Do they even understand what is happening on the reviewer side of the equation? I think some of the more clueless imagine that reviewers are just queuing up to write negative reviews about them and it is all the editors’ fault for not organizing things better.

My purpose in posting is to open the discussion. I think what is needed are some ground rules that would help the senior scholar problem. (1) Reviewer time is a scarce resource. Treat it as such. Do not waste people’s time. (2) No article is ever sent to more than three reviewers. Better is to send to two and ask for a third if there is a split vote.  (3) If a reviewer fails to respond in a timely fashion, they get an email: please respond or we will send the article to someone else. (4) If an editor has three reviews, they immediately send a notice to anyone else they asked for a review saying “we have enough now” or, if you insist, “we have three reviews but they are mixed, and your opinion would help.” (5) If you get two reviews and the situation is obvious, tell anyone else you asked for a review “never mind.”  (6) An R&R is sent back to the original reviewers and to NOBODY ELSE unless there is some very specific issue and the paper author is told at the time what the issue is and the category of additional reviewer who will be solicited. (7) Author angst about turn-around time is dealt with not by sending articles out to eight possible reviewers (!!!!) but by keeping authors informed of their status. Telling an author that they are having a hard time getting reviewers lets them know what is going on. (8) Tell reviewers you want a response to the “will you review?” email within two weeks and cancel the invitation if they do not respond within one week to the follow up to the initial request. Leaving the requests open just encourages the kind of gaming I described and increases the risk of wasting reviewers’ time with too many reviews.

To expand the pool of reviewers among junior scholars, it seems to me that there needs to be a database set up of potential reviewers. This would have to have cvs and samples of the person’s own publications/writing. Does anyone have an idea about how to get such a thing going?

Scatterplotters: Your thoughts?

Advertisements

Author: olderwoman

I'm a sociology professor but not only a sociology professor. It isn't hard to figure out my real name if you want to, but I keep it out of this blog because I don't want my name associated with it in a Google search. Although I never write anything in a public forum like a blog that I'd be ashamed to have associated with my name (and you shouldn't either!), it is illegal for me to use my position as a public employee to advance my religious or political views, and the pseudonym helps to preserve the distinction between my public and private identities. The pseudonym also helps to protect the people I may write about in describing public or semi-public events I've been involved with.

4 thoughts on “Too Many Reviewers”

  1. Dear older woman: I agree with you 100%. I think we need to storm the barricades (aka the annual editorial board meetings of AJS, ASR, SF, SPQ, etc.) & make the case there. But that’s almost a year away. In the meantime, I’m taking individual action. Here’s the screed I sent to the editors of ASR, along with my most recent review for them. I put it in the “comments for editors” box.

    I am willing to review a revision of this manuscript – but only on the condition that you do not send the paper to new reviewers on the second round. I’m fed up with the norm in sociology journals of adding new reviewers on subsequent rounds of review.
    When you editors bring in new reviewers on subsequent rounds (you are by no means alone in doing this, but you’re my target right now because it’s you I’m reviewing for), you violate the implicit contract that you create between authors and reviewers when you assign reviewers to a paper and transmit the reviews to authors for their consideration and response. I take the reviews of my own papers very seriously, just as I take the reviews that I write for journals very seriously. Bringing in new reviewers on the second round (or third or fourth…) disrespects both parties to the implicit contract you created when you selected reviewers for the first round.
    And every time you bring in new reviewers, you doom authors to new critiques of their papers. Sociology is a low-paradigm discipline, so different reviewers are bound to have different perspectives on any paper they read. Consider this analysis of a nearby discipline, psychology. In a 1990 article in American Psychologist, titled “But the reviewers are making different criticisms of my paper!”, Donald Fiske and Louis Fogg analyzed reviews of papers in psych journals and found very, very, very low correspondences between pairs of reviews in terms of the types of criticisms raised; they also found very, very, very low associations between pairs of reviews in terms of the overall evaluation (positive, neutral, or negative) of the article. Given the high probability that different reviewers will find different things to love and hate in a paper, it’s not at all surprising that the new reviewers who are brought on during the second round of review raise entirely novel concerns. This is what leads to the endless treadmill that many of us feel we’ve been on with some journals lately – including the journal you edit.
    So, to sum up, I am willing to review this paper on a second round, but only if you do not bring in new reviewers – unless one of the original reviewers dies or refuses to do the review, and you feel you absolutely must have someone with his/her expertise look over the paper.

    I encourage you & the many others who feel the same way as us to do something similar — let the editors of the journals for which you review know, in no uncertain terms, that the norm of bringing in new reviewers every round is unacceptable. This is something that tenured faculty at least should have no problem doing.

    Yours,
    another “older woman”

  2. Thanks for your comments, here and on orgtheory. It looks like it is time for a rebellion, indeed. I agree with your sentiments. I’m also thinking about boycotting for-profit journals while I’m at it.

  3. For example, when you click on a legal blog you will most likely see something that looks similar to the image at the right.
    If you do not have time to watch the Emmys 2012 tomorrow night, you can follow along in the comments section below
    to find out all the 2012 Emmy winners and results.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s