In a previous blog post I mentioned the perils of big data. I can also use the paper from that post to talk about the publication process.  Today I received an email that this paper has just been published at International Studies Quarterly.  This is a very good journal and I am very excited to have people read this paper.

I had a little twitter exchange about this paper so I very quickly wrote up a little post.  Sorry if this is a bit half assed, but I am on leave with my 3 month old son.  I have very little time to do work…and even less to blog. And it is a Friday night.  Shooting from the hip.

But here is a quick rundown of my paper.

My paper won an award for the best paper presented in the American Political Science Association Political Economy section 2008 and was first sent to the American Political Science Review on January 10, 2008.  It was finally published in ISQ on September 13, 2013.

What took so long?  It was rejected by multiple journals, often for very fair criticisms.  But this paper, more than many other papers, had a bunch of near misses.

Here is my recollection (with the aid of emails) of the publication process:

Submitted APSR 1/20/08.  Rejected 5/13/08

Submitted IO 10/14/2008, Rejected 1/06/09

Submitted JOP 7/11/09, Rejected 11/23/09

Submitted AJPS  6/19/10, R&R 12/?/10, Rejected 8/19/2011

Submitted ISQ 9/30/11, R&R on 2/24/12, resubmitted 6/6/12

Accepted: 6/22/12

I swear I had another R&R in there, but I might have this mixed up with another paper that shared a similar fate. I can think of at least five papers that have gone through more than five journals.

In my previous post I mentioned that this was a very labor-intensive project.  It required me to apply for a special access to the data data, obtain a security clearance, and then fly to Washington DC to do the analysis, and then to do more analysis for each reviewer.  So in between each one of these rejects are a series of flights to Washington DC, analysis, rewriting and finally resubmitting.

I finally quit on future projects with this data when we had my first child two years ago.

It was a pretty painful experience.

I’m a tenured professor at a great institution (WashU) and I’m moving as a tenured professor to another great institution (GWU).  If anything, I think I have caught a lot of breaks in my career and have encountered lots of reviewers that gave me more constructive feedback than I could have ever expected.  I have few complaints about how things have worked out for me.

But I think the point is that you can often look at the CV of a person and you only see the published papers and don’t see any of the failures along the way.  This paper is definitely on the long side from writing to publication, but the process rarely is quick and clean.  Here are a few quick facts.

• My most cited paper (a 2003 International Organization paper) went through 3 rounds of Revise and Resubmit.
• I have submitted three career NSF grants and one cross-division grant that were rejects.  I think I’m at seven or eight other rejected regular grants. NSF 12 Nate 0
• In the last two weeks I received two rejections from the American Political Science Review.  I have also received two Reject and Resubmits from the APSR in the last three years.  Both of these were rejected on the second go round as well.  Not sure of my total APSR rejects, but I know the accepts are zero.
• I have publish over 25 papers and I can only think of three that hit on the first submission.  I would guess that the modal number of journals before publication is three.  But that is a guess.

These are stories from mostly my own papers.  I’ve had some other crazy experiences with my co-authored papers.  This includes what seemed to be something like six rounds of R&R with a journal, a three line review after 9 month of waiting (basically saying that they didn’t like this kind of work), and a few projects that simply died in the publication process.

What are my own person conclusions from this process?

1. Editors have a lot of power, but they tend to be fair.

In my many, many rejected publications, I have only complained to the editor once.  This was a case where a reviewer basically vetoed the paper saying that they didn’t like it, but without any real explanation.  This was the pivotal review and my paper was rejected.  I complained to the editor  and the editor agreed to go out for a new review.

But other than that, I have found that most editors and reviewers are fair.  I often disagree with some of the points in a review, but I can’t think of a case where an editor has unfairly read the reviews and rejected my manuscript.  It is often a judgment call, and sometimes it has gone my way.  Probably the majority of times it went the other direction. But I can’t call it unfair.

I actually have a lot of sympathy for editors given the volume of submissions, the difficulty of getting good reviewers, and the complexity of making a decision of accept, reject, or R&R on some many difficult cases.

2. Sloppiness kills

As I noted above, I think most reviewers do their job.  But you can really, really piss off a reviewer in the first few pages.  This can be a typo, an incorrect reference, or a flashy intro that oversells your contribution.  There are lots of ways to set off a reviewer, and I can’t say for certain if this is why a review turns nasty, but I can say it is strongly correlated.  Ironically, this post probably has a bunch of typos.  Not peer reviewed.

3. Work on multiple projects…

I would often ask senior job candidates and speaker series visitors about their most famous projects.  Did they think it was their best work?  In general, can they gauge where a piece will land?  Most of these people were surprised what stuck and what didn’t.

This isn’t to say that quality isn’t correlated with where a paper lands, but there is a lot of uncertainty even with the strongest paper.  If it gets published, does it get cited?  I think this is even more of a mystery.

My personal response to this uncertainty is to diversify.  This also fits my own personality and interests.  I love working on and finishing projects, often with co-authors.  Obviously this isn’t a strategy that works for all types of research.

4.But…

There is some tradeoff between quantity and quality.  This seems so obvious that it isn’t worth mentioning. But, I’m not sure how much better my work would be if I cut the number of papers I write in half.  There are tradeoffs you have to make, but remember that even the most famous people have trouble telling ex ante what is their best work.  Make sure you have a real diamond that has been vetted by others if you’re going to polish and polish and polish. This is especially important for grad students who want to have a great project.  A great project doesn’t emerge from sitting in a room working and working and working.  It comes from writing, feedback, failure and trying again.

5. And…

It can be hard to manage workflow when you have a lot of papers floating around. This gets even more complicated with co-authors that are also juggling multiple projects.  It is very easy to ignore a project and let it sit for a year while you work on other projects.  For some people it might be better to work on sequential projects finishing one and then moving on to the next.

I work on multiple projects at the same time.  Mostly because I enjoy it, but I also have found some ways to really manage my workflow.  The most obvious of these is to make sure to get projects done and out the door.  The other is to make sure to make the best use of “downtime”.  When I have some free work hours, I write.

6. Most importantly…

Take reviewing manuscripts seriously.  Like most senior scholars, I get 20-40 manuscripts to review a year.  Add to this tenure reviews, papers by grad students, conference discussant duties and I’m sure I end up commenting on at least one research paper per week of the year.  Maybe two.

But these reviews really matter.  In a few cases I have taken to complaining to editors if I think another one of the reviewers has been unfair to a paper that I reviewed.

I’m not sure if these complaints matter and I am as guilty as anyone of some sloppy reviews.  But reviews matter.

This is a reminder to myself.  Reviews matter.

DISCLAIMER:  This is a blog post I whipped up in an hour or so.  It is rough and probably make some pretty stupid points.  I am tempted to delete it and not post it until I have a more polished version and some clearer thoughts.  But I have done this a few other times, only to never get back to the post.  So I am taking my own advice.  I’m going to throw a few ideas out there and see what sticks.  See what people say about them, and use this feedback to revise my own thoughts.  The only advice I didn’t take is the one on sloppiness.  Sorry for all of the typos Mom.