A few weeks ago I shared a story of the paper that was published after over five years in the pipeline.  This wasn’t my best or worst publication experience.  I told this story because it was being published online the day of my post.

I was shocked at the number of people who viewed the post.  Google Analytics (with no IP tracking) has it at almost 8,000 views.  I’ve heard from many faculty members that my experience isn’t uncommon.  The fact that so many people viewed my post probably means that most of us just don’t talk about these failures.

I have another story.  This one might be a little more unique.  Almost every faculty member I know has quit on at least one research project.  Often this is when the “market” (reviewers) send a clear signal that the paper won’t be published.  Anywhere.

I had one paper that was constantly caught between American, Comparative, and International Relations reviewers.  They all though it was ok.  None of them wanted to publish it.  This is a warning about sitting at the intersection of a couple of fields.  Your paper will sit.  Maybe forever.

But the story I am going to tell is different. It is also about quitting a research project after many years and many, many hours of work.  But it is also about research ethics. I don’t know if I made the ethical choice.

My previous blog post centered around a paper on politics and corporate taxes that was published at International Studies Quarterly.  This tax paper used confidential data from the US Bureau of Economic Analysis (BEA).  As I documented in my another post, the start-up costs are high.  And so are the marginal costs.  I had to get a security clearance, learn the data, merge and clean the data, and then run every single regression in Washington D.C.  But I finally published a paper from this data.

The original plan wasn’t to get a single paper out of this high cost project.  I had another piece that I wrote using this data.   I presented this paper at multiple conferences, used the preliminary results to submit an unsuccessful NSF grant, and (rejected) journal submissions.

Before I introduce my dilemma, here is some very quick background.  This paper was started when I was a junior faculty member at Washington University.  My department conducts annual reviews for junior faculty members.  Junior faculty submit their vita, working papers, and even some questions for the senior faculty members every year.

I was lucky enough to get out of the gates quickly and have a couple of papers that were published in grad school and in my first few years as an assistant professor.  I worked on turning my dissertation into a book while also thinking about follow-up projects ranging from economic voting, trade policy, to civil war.  This eclectic mix of topics sounds pretty silly right now, but as a second year faculty member I was simply following my interests.  Good thing we had an annual review.

The senior faculty gave me some very good advice.  First, when I go up for tenure, there should be a set of reviewers that know most of my work. Second, I should do my best to be known for something.  Not just a paper or an idea, but a body of work.

I decided to focus most of my work, but not all of it, on the study of politics and foreign direct investment.  My paper that was eventually was published at International Studies Quarterly was a new project on firms and taxation.

But I also had a very clear idea for an extension of my dissertation work.  My dissertation focused on how political institutions shape the risk environment for firms and the BEA has fantastic firm level data that I would use to test some of the theories from my dissertation in a much more vigorous way.

This post is already too long, so let me skip to the good part.  After years of working with the data at the BEA, presenting at conferences and ultimately going through the review process with a rejection or two, I quit the project.

In my paper I found that firms around the world behaved substantially differently under democratic regimes than non-democratic regimes.  Forget about the fact that I was looking at firms that had invested in a country and that there is a serious selection problem.  I told you to forget about that part.  Where you Reviewer #2?

My reviewers highlighted the obvious research design problems, but were willing to overlook some of these issues based on the novelty of the project. They also seemed to recognize the complexity of trying to model the selection decision of firms, given that I had 20,000 firms investing in 200 different countries, and I had to do all of my analysis on location in DC.  The reviewers were much, much more forgiving of the limitation of what I could do.  Except Reviewer #2.

My friends would ask me about this paper.  Some of them would call it the “underwear paper”.  The BEA requires researchers to conduct all of the analysis in a windowless, secure room and I joked during a conference presentation that I couldn’t bring anything into the room and had to run my regressions in my underwear.  Sorry for that mental picture.

A few commentators at conferences mentioned that I focused too much on the statistical significance of my results and this could just be an artifact of the sample size.   A methodologist colleague thought I shouldn’t have standard errors at all.  This is the population.  In all, the substantive significance of my results was less impressive than the stars in my regression tables.

Every five years the BEA collects a firm census and new data became available in the middle of this project.  This new data gave me the opportunity to test my results with another year of data.  It wasn’t really necessary, since most of the reviewers weren’t asking for more data.  But I had access to the data and I eventually wanted to use more years, so why not test the robustness of my results?  I took another flight to DC, spent a few days preparing the data and I finally ran my models.

My findings weren’t robust. Forget about the technical details.  I no longer could in good faith say that I believed my original findings.

I told myself that a less ethical junior faculty member would forget about the new data and just kept pushing the old paper.   But, I decided that I could no longer send out this paper for review.  In reality, I really don’t think my colleagues would try to publish something that they know is clearly wrong.   Most people would have done the same in my position.

Or maybe not.  Over the years I have been paying more and more attention to the studies of publication bias.  It’s often easy to imagine some immoral researcher running regressions until enough stars pop up.  Did I do the opposite?  Did I drop a project because it didn’t give a clean result I could push into a journal?

I honestly haven’t though about this paper too much over the years.  I moved on.  But what I can say is that publishing is hard.  Damn hard.  Even the best paper that gets accepted at the first journal is a slog.  Reviews come back and you need a combination of a thick skin, belief that your project is important, and some optimism that this paper will make it through the capricious review process.

I’ve developed a pretty thick skin over the years, and I try to pick projects that matter.  At least to me.  It is the optimism that is the hard part.  For every paper you have to finish up the last 5% and drag the work across the finish line.  It is the least fun part of the process and makes me so sick of a paper that I often don’t read it again after it is published.  But this last part is essential, and probably where many people get hung up.

With a problematic research design and now contrasting results, I couldn’t justify the financial cost of going to DC a few more times and to do conduct days of non-stop data analysis.  I now have two little kids and let my BEA access expire.  The travel time is too much for my family.

I knew that this would be a really tough paper to ever get published, and I decided that I would focus more on new projects.  Was I going to spend a few more thousand dollars and countless hours running regressions in my underwear in the hope of sorting this out?  Sorry again.

My story is one of staying true to only publishing what you are confident in, while at the same time contributing to publication bias in the discipline.  Actually, this paper was really never going to make it any way.  My counterfactual reviewers are to blame.  Are my hands clean?

Forget about ethics.  Let’s go back to my previous post.  I have one dropped project and one project that took over five years to get through the publication process.

The longer you are on the profession the more a second truth becomes self-evident. Publishing is hard.  Damn hard.  It is easy to look at a person’s CV and be intimidated by their success.  But don’t think that this comes easy for anyone.

I am an advocate of judging the written work for junior hires and the published work at tenure time.  I probably weigh letters, reputation, or even the job talk less than many others.  I say we should read the working papers and look at the published work.  But this doesn’t mean that we have to use labels like “deadwood associates” or mock junior faculty that don’t get a paper published in their first few years on the job.  Don’t assume failure is caused by a lack of effort.

Also, admitting how hard this process is shouldn’t demoralize graduate students just entering the profession.  The best published scholars in the discipline send out lots of papers, deal with rejection, revise, get rejected, revise.  They rethink projects, publication strategies, and go through some soul searching in the process.  You can learn from them.  Ask your advisor about this.  It is hard for everyone.  But it can be done.