Too Many Reviewers

(reposting here to archive. Discussion is like to be on scatter.wordpress.com)

I freaked out recently when, after reviewing an article, I received a packet of FIVE (5!!!)  reviews on the same article. I chewed out the editors for wasting my time and told them I would never review for their journal again. After an exchange (in which I got a little less testy), I told them I’d post my concerns to scatterplot and open a discussion on the topic. Although five was over the top and freaked me out, it has become pretty common now for me as a reviewer to get a packet with four reviews. No wonder we regular reviewers are feeling under the gun. The old calculation of two or even three reviews per article has gone by the wayside. The pressure for fast turnaround and the high turn-down or non-response rate among potential reviewers has led editors to send out articles to extra reviewers in the hopes of ending up with at least the minimum two or three.

But this is a death spiral. As a frequently-sought reviewer I get at least four requests a month, sometimes as many as eight, and I used to get more before I got so crabby.  When I was young and eager, I was reviewing an article a week [and thus, by the way, having a huge influence on my specialty area], and I know some people who are keeping that pace. But at some point you burn out and say “no more.” I, like all other frequently-sought reviewers I know, turn down outright the requests from journals I don’t know for articles that sound boring, and then save up the other requests and once a month pick which articles I want to review. So the interesting-sounding articles from good journals get too many reviewers, while the boring-sounding articles from no-name journals get none. If journal editors respond to the non-response by reviewers to boring-sounding articles by sending out even more reviewer requests per article, our mailboxes will be flooded even more and the non-response rate and delayed-response rate by reviewers will go up even more. Senior scholars are asked to review six to eight (or more?) articles per month. You have to say no to most of the requests.

And then we have the totally out of hand R&R problem. I think it is completely immoral to send an R&R to ANY new reviewers. I know a young scholar with a perfectly good paper who is now on the 4th (!!!!) iteration of an R&R from ASR. Not because she has not satisfied the original reviewers, but because the editors keep sending each revision to a new set of reviewers in addition to the original reviewers and, of course, the new reviewers have a different perspective and a new set of suggestions for the paper, some of which cover ground that was gone over in one or more of the previous revisions. Not to mention the problem that R&R memos are now longer than the original articles!!  We are no longer a discipline of article publishing, we are turning into a discipline of R&R memo-writing.

Something has to change.  Senior scholars burn out and get reputations for being difficult, possibly because editors don’t know how many other people are asking them to do things. Junior scholars would want to review and wonder why nobody is asking them, and other junior scholars think they are being tapped a lot because they are getting four requests a year. Article-submitters (disproportionately junior scholars) whine and complain about slow turn-arounds, and imagine — what?  I guess I don’t know what they imagine? Do they even understand what is happening on the reviewer side of the equation? I think some of the more clueless imagine that reviewers are just queuing up to write negative reviews about them and it is all the editors’ fault for not organizing things better.

My purpose in posting is to open the discussion. I think what is needed are some ground rules that would help the senior scholar problem. (1) Reviewer time is a scarce resource. Treat it as such. Do not waste people’s time. (2) No article is ever sent to more than three reviewers. Better is to send to two and ask for a third if there is a split vote.  (3) If a reviewer fails to respond in a timely fashion, they get an email: please respond or we will send the article to someone else. (4) If an editor has three reviews, they immediately send a notice to anyone else they asked for a review saying “we have enough now” or, if you insist, “we have three reviews but they are mixed, and your opinion would help.” (5) If you get two reviews and the situation is obvious, tell anyone else you asked for a review “never mind.”  (6) An R&R is sent back to the original reviewers and to NOBODY ELSE unless there is some very specific issue and the paper author is told at the time what the issue is and the category of additional reviewer who will be solicited. (7) Author angst about turn-around time is dealt with not by sending articles out to eight possible reviewers (!!!!) but by keeping authors informed of their status. Telling an author that they are having a hard time getting reviewers lets them know what is going on. (8) Tell reviewers you want a response to the “will you review?” email within two weeks and cancel the invitation if they do not respond within one week to the follow up to the initial request. Leaving the requests open just encourages the kind of gaming I described and increases the risk of wasting reviewers’ time with too many reviews.

To expand the pool of reviewers among junior scholars, it seems to me that there needs to be a database set up of potential reviewers. This would have to have cvs and samples of the person’s own publications/writing. Does anyone have an idea about how to get such a thing going?

Scatterplotters: Your thoughts?

Sociology and Spirituality

From Jay MacLeod Ain’t No Makin’ It, third edition, p 504 (the last paragraph of the methodological appendix):

When I visited Chris in prison and he asked me about my faith, I suggested that spirituality can arrest our inertial drift into self-deception.  My faith in a forgiving god allows me to face up to the truth about myself and to deal constructively with my sin.  The United States is even more prone to self-deception than I am.  We are in the grip of denial and resistance to the reality of our social sin, and sociology can help the world work through its ignorance of itself.  Spirituality and sociology have parallel vocations. Spirituality reveals the truth about ourselves. Sociology reveals the truth about our society. Both spur us to struggle for justice, for in the end my redemption is linked to yours.

This resonates for me.

Bibliographic note: MacLeod wrote what became the first edition of Ain’t No Makin’ It as an undergraduate thesis; he went on to be a community organizer and then an Anglican priest.  In case you don’t know the book, the first edition was based on observation and interviews with poor White and Black boys in 1983, the second edition caught up with how they were doing in 1991, and the third edition finds them in 2007.    The popular hook in the first edition was that the “Hallway Hangers” whose lives centered on substance abuse and crime were mostly White, while the “Brothers” who avoided misbehavior and tried hard in school were mostly Black, so the discussion of the impact of class and structural constraints ran against some of the usual grains. The “where are they now” follow ups pull this book apart from most in the genre.  It would be a good book to teach from. Probably the most useful policy implication is MacLeod’s argument that poor youths — and adults — should know the structural constraints they are up against if they are to avoid self-blame, despair, and self-destructive behavior.

Update:  Here is the publisher’s page for the book.  Most of the “hits” for this title in Google are to term paper vendors. Watch out.

Social Psychology Conference

Glen Elder Life Course Theory

Tape-delayed blogging of the social psychology centennial conference held at Wisconsin Sept 26-7 including talks by Glen Elder, Shelley Correll, Mitch Duneier, Yuri Miyamoto, Terri Orbuch, and Jim House.  This conference was  honor of the first publication of books with the title Social Psychology, one of them by E.A. Ross, a founder of the  Wisconsin sociology department.  This conference is held in the room that is not named after E.A. Ross; the not-naming occurred after a two-hour debate in the early 1980s about whether the racism of Ross’s “race suicide” Social Darwinist work outweighed his support for working people and his belief that sociology should address social problems.  I arrived late, after the administrative welcomes and most of the way through John DeLamater’s summary of the history of social psychology.

Glen Elder talked about doing longitudinal life course research.  A lot of the talk was anecdote about his research career.  The point where many of us started taking notes was this graphic.  Although there were questions about what he means by “theory,” to which Elder replied that he means “orienting concepts” or “framework,” I was struck by how apt this graphic was as a representation of what I feel I’ve learned about living life.

Friday evening we had a fancy dinner and then watched the presidential debate on the big screen.  The social psychologists yelled and booed like they were at a wrestling match, and I would have felt pretty uncomfortable  if I’d been a Republican.  We were in the Business School’s conference center, and there were business folks watching the debates on TVs in other rooms; I wondered if the atmosphere would be different if we’d mingled with other groups.  The other highlight of the evening for me occurred earlier: after I described the troubles we were having finding a place to hold my daughter’s wedding reception in December, a prominent psychologist had me about falling on the floor laughing as she advocated renting the zoo for the reception.  As she said, the zoo really needs the business, it would probably be cold enough for the pond to freeze so you could dance on it, and if it was really cold you could go into the lion house to get warm.

Saturday morning there was a little flurry around miscommunication.  People were carrying in their cups of Starbucks and bags of Einstein Bagels, only to discover that there was enough coffee and bagels in the conference room for about twice as many of us as were there.  So at the end of the day, everyone was urged to take bagels home for the freezer.

Shelley Correll led off the Saturday events, talking about the value of experiments with an emphasis on gender research.  I was particularly struck by the “motherhood penalty” research  (Correll, Shelley J, Stephen Benard, and In Paik. 2007. “Getting a job: Is there a motherhood penalty?” American Journal of Sociology 112: 1297-1338.)  I guess I wasn’t paying attention last year when this article came out: experimental results that show that people with exactly the same credentials are more negatively evaluated if there is one line on the c.v. that she is a mother, and that taking time off work for going to the gym is viewed more positively than taking time to have lunch with a child.   (Fathers experience no penalty – this is coming back to me.)  She also cited experimental results by someone whose name I missed that men harassed more by sending pornography to women if their masculinity was threatened.

Mitch Duneier began by making laudatory remarks about a lot of people (many of them in the room) and Wisconsin sociology and stressing that he is not a social psychologist although he appreciates and learns from social psychology.  His main talk was a discussion of the danger in qualitative methods of cherry-picking, that is, of finding the quotations or incidents that support your point.   He also talked about ethics and the danger of treating IRB approval as the end rather than the beginning of ethical behavior, and worrying about the problem of using other people for our purposes and befitting at their expense.  Good talk, but a safe talk, nothing to disagree with, the only overt target of critique is himself.

Yuri Miyamoto is a psychologist who presented a variety of experimental results contrasting the modal response of Easterners (Japanese or Chinese) with Westerners (Anglo-Americans), where Easterners tend to think more about context, locate causes in relations, and favor aphorisms stressing change and contradiction, while Westerners prefer analytic thinking & laws of non-contradiction and locate causality in objects.  The details are interesting.  When describing a picture do you start with the fish – the focal objects – or do you start with context e.g. “this is a lake bottom.”  When told an essay was written by a student who was required to argue for a given side, do you attribute its opinion more to the writer if it is long, complex and passionate versus short and perfunctory?  (Westerns don’t distinguish, Easterners do.)  Which two of “chicken, cow, grass” go together?  Westerners tend to say chicken & cow (both animals), Easterners say cow & grass (cow eats grass).  She suggests that these differences arise from settlement patterns, the longer history of dense settlement and highly interdependent agricultural life in Asia vs. frontier individualism in the US.  Research in Hokkaido, the northernmost island of Japan not settled until the 19th Century, finds that ethnic Japanese there give answers more like Americans.   Sociologists in Q&A and conversation during break comment critically on how all difference is packed together as under-theorized “culture” and the importance of looking at variation within groups.  Good points, but I still found the main axis of the story to be worthy of contemplation.  I’ve thought before about how the US is fundamentally the place dominated by people who left their families and roots behind.

Terri Orbuch AKA “The Love Doctor” gives a fact-filled summary of the research on how people come to have happy marriages versus unhappy marriages or divorces, organizing her talk around the three main strands of social psychology: symbolic interaction, the dynamics of group interaction, and the social construction of meaning.  Most of the talk summarizes results from the longitudinal study of newly-married couples.  If you are just starting out on the coupled life, the best predictor of long-term stability for everyone is “affective affirmation” – the overt communication of loving attitudes.  Interestingly, “cognitive collaboration,” developing a joint narrative of your life, is a secondary positive predictor of stability for Whites but negative (although non-significant) for Blacks.  Divorce rates are higher for Blacks than Whites, controlling for everything they could control for.  Another big racial difference: Black men do much more housework, feel less threatened by involvement at home,  and have much more egalitarian gender ideologies than White men; Black women’s marital happiness depends on how much work at home their husbands do, while White women’s does not.  Men’s work at home is a predictor of divorce for Whites but of non-divorce for Blacks.  These racial differences in the gender attitudes of Black people are consistent with other research I’ve seen years ago, and tend to go unnoticed by Whites, except perhaps for feeding into stereotypes of Black women as emasculating.

Jim House gave a shorter re-presentation of his 2007 Cooley-Mead lecture.  The first part of the talk is the history of social psychology, showing how central and important it was through the first two-thirds of the 20th century, and then how it has declined since 1970 in terms of section memberships and proportion of top departments with a specialty in social psychology.  (Social psychology has also declined within psychology.)  He argues that microeconomics filled that space.  This is due both to ideological factors, the rise in neolibral individualism and the valorization of business firms, and (relatedly) to deep cuts in the governmental funding support for social science in the 1980s, and the relative insulation of economics from these cuts.  Tversky/Kahneman’s social psychological work was translated into and published in economics journals, and won the Nobel prize in economics.  Economics has replaced social psychology in public policy.  In addition to the ideological context argument, House argues that the expansion of sociology and the proliferation of sections paradoxically weakened sociology as compared to economics, which has no sections in the AEA and rather rigidly enforces ideological hegemony.  All academic fields rose 1940-1970, then generally flattened out.  But after 1970, total number of economists declined somewhat, while the number of sociologists grew modestly.

To save social psychology from decline, House stresses path dependence and the role of human agency.  He says social psychologists need to act, to deal publicly and aggressively with the problems & limitations of economic models.  In light of economic crises, he talks about how social psychologists have a lot of knowledge they should share about how and why people fail to fit economistic decision-theory models.  There is a need to balance rational choice with understandings of structural constraint and non-rational motivation/values.  And social psychologists need to link with social, biomedical, and natural sciences.  He also says we should reduce hyper-specialization, reduce the number of new PhDs who know only about their own applied interests.  He contrasts this with the sciences in which everyone has to have a background in physics, chemistry, biology at least at basic level.  I found the specialization argument misspecified, and talked to him about it afterwards.  In his accounts, economists are even more ignorant of other fields than sociologists are, and the interdisciplinary training he lauded about the physical and biological sciences happens in the undergraduate curriculum.

Vigorous discussion ensued and was halted 45 minutes after the announced ending time by the growing trickle of people leaving to catch their planes.  Good event.

one last asa gripe

I’ve just learned that I messed up.  I was supposed to choose WHICH roundtable session to forward those incoming papers to .  According to Kendra at ASA, I should have known this because last October I received an email that says “second priority organizers are responsible for forwarding unused acceptable papers to the appropriate roundtable organizer.”  Apparently just clicking reject did not automatically forward the paper (unlike the first round process), and apparently submitters don’t get to choose their roundtable back up, their second choice session organizer is supposed to make this choice for them.  Bad word bad word bad word.  I sent a complaint pointing out that this system makes no sense, as it is the paper submitter who ought to choose which roundtable session to go to, and that important instructions should be posted on the organizer web site, not hidden in old email messages four months removed from when and where they would be needed.  I also alerted Kendra at ASA to this problem so she can hopefully find the lost papers and put them in the right place.  Warning to everyone whose paper got “rejected” rather than forwarded to a roundtable: this is probably what happened.   It appears that Kendra at ASA is watching for orphan papers that need to be forwarded to roundtables, so I think she is the person to talk to if you got messed with in this way.  If you are mad, please curse your session organizer for inability to decipher a cryptic and counter-intuitive system, not an indifference to your career.

asa session organizer site

Quick question.  ASA’s session organizer site appears non-functional.  First no response, then administrator’s login where it has never heard of me.  Anybody know what is going on?  ASA isn’t answering email; I did not try calling.  (I’m almost done, just need to forward 1 paper to round tables if it gets released by its first choice.  I can see why people don’t remember to do this.)

dysfunctional ASA rules?

I like on-line paper submission a lot. But I’m worried that the rigid selection rules ASA has imposed along with the on-line system are dysfunctional, specifically that a paper can only be seen by the organizer to whom it has been submitted and that organizer of choice #2 only sees the paper if organizer of choice #1 rejects it. Back in the bad old days, we had to deal with boxes and boxes of paper — I got several hundred submissions one year —