Too Many Reviewers

(reposting here to archive. Discussion is like to be on scatter.wordpress.com)

I freaked out recently when, after reviewing an article, I received a packet of FIVE (5!!!)  reviews on the same article. I chewed out the editors for wasting my time and told them I would never review for their journal again. After an exchange (in which I got a little less testy), I told them I’d post my concerns to scatterplot and open a discussion on the topic. Although five was over the top and freaked me out, it has become pretty common now for me as a reviewer to get a packet with four reviews. No wonder we regular reviewers are feeling under the gun. The old calculation of two or even three reviews per article has gone by the wayside. The pressure for fast turnaround and the high turn-down or non-response rate among potential reviewers has led editors to send out articles to extra reviewers in the hopes of ending up with at least the minimum two or three.

But this is a death spiral. As a frequently-sought reviewer I get at least four requests a month, sometimes as many as eight, and I used to get more before I got so crabby.  When I was young and eager, I was reviewing an article a week [and thus, by the way, having a huge influence on my specialty area], and I know some people who are keeping that pace. But at some point you burn out and say “no more.” I, like all other frequently-sought reviewers I know, turn down outright the requests from journals I don’t know for articles that sound boring, and then save up the other requests and once a month pick which articles I want to review. So the interesting-sounding articles from good journals get too many reviewers, while the boring-sounding articles from no-name journals get none. If journal editors respond to the non-response by reviewers to boring-sounding articles by sending out even more reviewer requests per article, our mailboxes will be flooded even more and the non-response rate and delayed-response rate by reviewers will go up even more. Senior scholars are asked to review six to eight (or more?) articles per month. You have to say no to most of the requests.

And then we have the totally out of hand R&R problem. I think it is completely immoral to send an R&R to ANY new reviewers. I know a young scholar with a perfectly good paper who is now on the 4th (!!!!) iteration of an R&R from ASR. Not because she has not satisfied the original reviewers, but because the editors keep sending each revision to a new set of reviewers in addition to the original reviewers and, of course, the new reviewers have a different perspective and a new set of suggestions for the paper, some of which cover ground that was gone over in one or more of the previous revisions. Not to mention the problem that R&R memos are now longer than the original articles!!  We are no longer a discipline of article publishing, we are turning into a discipline of R&R memo-writing.

Something has to change.  Senior scholars burn out and get reputations for being difficult, possibly because editors don’t know how many other people are asking them to do things. Junior scholars would want to review and wonder why nobody is asking them, and other junior scholars think they are being tapped a lot because they are getting four requests a year. Article-submitters (disproportionately junior scholars) whine and complain about slow turn-arounds, and imagine — what?  I guess I don’t know what they imagine? Do they even understand what is happening on the reviewer side of the equation? I think some of the more clueless imagine that reviewers are just queuing up to write negative reviews about them and it is all the editors’ fault for not organizing things better.

My purpose in posting is to open the discussion. I think what is needed are some ground rules that would help the senior scholar problem. (1) Reviewer time is a scarce resource. Treat it as such. Do not waste people’s time. (2) No article is ever sent to more than three reviewers. Better is to send to two and ask for a third if there is a split vote.  (3) If a reviewer fails to respond in a timely fashion, they get an email: please respond or we will send the article to someone else. (4) If an editor has three reviews, they immediately send a notice to anyone else they asked for a review saying “we have enough now” or, if you insist, “we have three reviews but they are mixed, and your opinion would help.” (5) If you get two reviews and the situation is obvious, tell anyone else you asked for a review “never mind.”  (6) An R&R is sent back to the original reviewers and to NOBODY ELSE unless there is some very specific issue and the paper author is told at the time what the issue is and the category of additional reviewer who will be solicited. (7) Author angst about turn-around time is dealt with not by sending articles out to eight possible reviewers (!!!!) but by keeping authors informed of their status. Telling an author that they are having a hard time getting reviewers lets them know what is going on. (8) Tell reviewers you want a response to the “will you review?” email within two weeks and cancel the invitation if they do not respond within one week to the follow up to the initial request. Leaving the requests open just encourages the kind of gaming I described and increases the risk of wasting reviewers’ time with too many reviews.

To expand the pool of reviewers among junior scholars, it seems to me that there needs to be a database set up of potential reviewers. This would have to have cvs and samples of the person’s own publications/writing. Does anyone have an idea about how to get such a thing going?

Scatterplotters: Your thoughts?

Sociology and Spirituality

From Jay MacLeod Ain’t No Makin’ It, third edition, p 504 (the last paragraph of the methodological appendix):

When I visited Chris in prison and he asked me about my faith, I suggested that spirituality can arrest our inertial drift into self-deception.  My faith in a forgiving god allows me to face up to the truth about myself and to deal constructively with my sin.  The United States is even more prone to self-deception than I am.  We are in the grip of denial and resistance to the reality of our social sin, and sociology can help the world work through its ignorance of itself.  Spirituality and sociology have parallel vocations. Spirituality reveals the truth about ourselves. Sociology reveals the truth about our society. Both spur us to struggle for justice, for in the end my redemption is linked to yours.

This resonates for me.

Bibliographic note: MacLeod wrote what became the first edition of Ain’t No Makin’ It as an undergraduate thesis; he went on to be a community organizer and then an Anglican priest.  In case you don’t know the book, the first edition was based on observation and interviews with poor White and Black boys in 1983, the second edition caught up with how they were doing in 1991, and the third edition finds them in 2007.    The popular hook in the first edition was that the “Hallway Hangers” whose lives centered on substance abuse and crime were mostly White, while the “Brothers” who avoided misbehavior and tried hard in school were mostly Black, so the discussion of the impact of class and structural constraints ran against some of the usual grains. The “where are they now” follow ups pull this book apart from most in the genre.  It would be a good book to teach from. Probably the most useful policy implication is MacLeod’s argument that poor youths — and adults — should know the structural constraints they are up against if they are to avoid self-blame, despair, and self-destructive behavior.

Update:  Here is the publisher’s page for the book.  Most of the “hits” for this title in Google are to term paper vendors. Watch out.

blog commenting, blinding articles

I’m in California helping my mother, who is much better than she was in March, but still mostly housebound, on oxygen, and weak. She’s easily frustrated and demanding, which is both understandable and tiring. One upshot is that I’ve been blog commenting all day. I do get periods of free time, but as I never know when I’m going to be interrupted, it is hard to crawl into the writing I’m way-overdue on. So far I’m 0 for 0 in my goal of working at least two hours a day, although I have gotten some reviews done.

Apropos of bits of work, here’s a question about articles for review. Say there is a pretty well-known data set that is identified with exactly one research team, such that anybody who has read the literature will recognize the authors, or at least the PI, from the data. It does not matter whether you use “identifying reference withheld” or third person references, the author, or at least the author’s research team, will be identified. How much trouble should the author go to in attempting to present the appearance of conforming to the norm of not identifying the author, i.e. in using the third person in describing the research procedures? (Identifying reference withheld would seem absurd in this situation, especially as the withheld citations would be central to any literature review.)

My view as a reviewer: if the author of an article in my area is a senior person, I know who it is. But under the cloak of the anonymous reviewer, I am willing to say critical things about my friends. And good things about people whose work I don’t recognize (and thus know must be junior). I have been told by editors that this is common; people will not say bad work is good just because their friends wrote it. It is the anonymity of the reviewer that is central to the integrity of the process. So I don’t think trying to pretend anonymity is worth the trouble. In fact, I loathe “identifying reference withheld”!! If you are going to anonymize, do it with third person references.

What do you think about an author who does not even go through the motions of third person references to disguise authorship, in a case where disguise would be futile anyway. Is that bad form? Or understandable, let it go? Should editors police this kind of thing? Do they?

tenure and public sociology

There have been several posts lately about public sociology and the tenure process, including newsocprof and thepublicandtheprivate and, most recently RadioFreeNewport. All are written by young scholars. The general tone of these remarks is either to worry about the impact of public sociology on getting tenure, or to decry older sociologists who tell younger sociologists to focus on getting tenure before getting heavily involved in public sociology. So another view seems helpful. I say this as someone who did not do public sociology until later in my career, well after I had tenure. These younger writers are ignoring the central point that tenure protects you when you do public sociology.

one last asa gripe

I’ve just learned that I messed up.  I was supposed to choose WHICH roundtable session to forward those incoming papers to .  According to Kendra at ASA, I should have known this because last October I received an email that says “second priority organizers are responsible for forwarding unused acceptable papers to the appropriate roundtable organizer.”  Apparently just clicking reject did not automatically forward the paper (unlike the first round process), and apparently submitters don’t get to choose their roundtable back up, their second choice session organizer is supposed to make this choice for them.  Bad word bad word bad word.  I sent a complaint pointing out that this system makes no sense, as it is the paper submitter who ought to choose which roundtable session to go to, and that important instructions should be posted on the organizer web site, not hidden in old email messages four months removed from when and where they would be needed.  I also alerted Kendra at ASA to this problem so she can hopefully find the lost papers and put them in the right place.  Warning to everyone whose paper got “rejected” rather than forwarded to a roundtable: this is probably what happened.   It appears that Kendra at ASA is watching for orphan papers that need to be forwarded to roundtables, so I think she is the person to talk to if you got messed with in this way.  If you are mad, please curse your session organizer for inability to decipher a cryptic and counter-intuitive system, not an indifference to your career.

how to be a good advisee

As I suggested in response to the thread about picking an advisor, it is a mistake to view an advisor as a commodity for which you comparison shop, as you might select a new dress.  Rather, it is a two-sided process of building a long-term relationship.  Your own behavior and characteristics are just as important as the advisor’s, and it isn’t just a matter of finding the right person, it is a matter of acting in ways that make both of you feel good about your interactions.  So it is important to consider what makes the experience good for the advisor, not just what makes it good for the student.  In the long run, former advisees are friends and junior colleagues and part of your professional network.  Having former students who do well in the profession make you look good.  But there can be plenty of immediate rewards in the advising experience itself.  This varies somewhat depending on personalities, of course, and others may have other opinions.  But here are the things I think about when I reflect on advisees I have appreciated and advisees who have been less satisfactory.

asa session organizer site

Quick question.  ASA’s session organizer site appears non-functional.  First no response, then administrator’s login where it has never heard of me.  Anybody know what is going on?  ASA isn’t answering email; I did not try calling.  (I’m almost done, just need to forward 1 paper to round tables if it gets released by its first choice.  I can see why people don’t remember to do this.)