Redundant, Redundant, Redundant: Economic Development Incentives

 

I have taken a break from blogging in my move from George Washington University to the University of Texas. I’m back!  Hello?  Anyone there?

My previous research (with many co-authors including Eddy Malesky) has looked at the use of economic development incentives to attract jobs.  In two published papers we argue that politicians use incentives to take credit for economic development and this leads to the overuse of incentives.

My big criticism of incentives are they are often redundant.  Incentives are tax breaks or grants to induce firms to relocate, expand or stay put.  But academic research shows that the majority of firms would have made the same decision (relocate, expand, or stay) even without incentives.  In these cases, incentives are just a transfer of tax payer funded benefits to firms for no new economic activity.

Two stories this week reiterate this redundancy.

First, ConAgra chose to relocate from Nebraska to Chicago despite Chicago offering less than half of the economic development incentives.  A company taking a smaller incentive offer isn’t a smoking gun that incentives weren’t effective.  More telling are the CEOs statements saying that these were not pivotal in the investment location decision.

“The decision to move headquarters was solely based on the strategic needs of our business and was not a city-vs-city exercise.”

According to newspaper reports, this didn’t stop the company from claiming to government officials that incentives were necessary and used creative accounting to get around an Illinois policy freezing new economic development incentives.  According to the Omaha World Harold:

But documents newly obtained by The World-Herald also show that ConAgra officials told the Illinois state government a different story in the months prior to its announcement.

ConAgra told Illinois officials that tax incentives were needed to justify moving its offices to Chicago. Illinois officials must have been convinced. They found a way around a statewide moratorium on incentives the governor had recently imposed because of a budget crisis in Illinois.

In short, prior to moving the company did everything possible to maximize their incentive tax.  After the deal was struck the company claimed that these incentives weren’t necessary.

Second, Marriott international moved its headquarters to Bethesda, MD with $62 million in economic development incentives.  This is a big deal and the new economic activity for a corporate location could have a major economic development impact.  Where did they move from?  From four miles down the road in Maryland.

A short move like this isn’t a smoking gun that incentives weren’t important.  It is plausible that the company could have moved to DC or Virginia with the right deal.  What is more revealing is this paragraph from the Washington Post:

Officials in D.C. and Virginia discussed a pursuit of Marriott, but it’s unclear how aggressively they pushed. Leaders in both jurisdictions remained wary about chasing a company they viewed as likely to remain in Maryland, according to officials familiar with the process were not authorized to discuss it.

It certain sounds like incentives were redundant in both cases.

I have a short podcast at the Scholars Strategy Network on the topic if you are interested.

Our Pre-registered Field Experiment on Investment Incentives

Our latest project on economic development incentives is up for comments.

http://equitablegrowth.org/electoral-institutions-and-electoral-cycles-in-foreign-direct-investment/

This project is a field experiment with Mike Findley and Dan Nielson on how election timing shapes the use of local economic development dollars.  Our pre-analysis plan was registered at EGAP.

The Washington Center for Equitable Growth is circulating this as part of their working paper series. (They did not fund the research).

Comments welcome!

 

Teaching in a Business School

For twelve years I taught political science at Washington University in St. Louis. This is my second and last year of teaching International Business at George Washington University before going back to political science. I am joining the Government Department at the University of Texas in Fall 2016.

A few of my friends have asked me about the differences in teaching at a professional school and teaching in the arts and sciences. This is a difficult comparison but my general message is that business school teaching is pretty tough. But there are rewards.

Teaching Loads

When I started my first job at Washington University in 2002 we had a 2-2 teaching load (four courses per year). After a study of the teaching loads at peer political science programs WashU shifted to a 2-1 teaching load.

In business schools, some faculty teach 2-2 (or higher teaching loads) but most “research active” faculty teach a three course load. Unlike most political science departments, faculty sometimes teach all three courses in a single semester and this can be the same class repeated three times. Faculty can teach three sections of “Introduction to International Business” or “Global Strategy” in a semester and not teach the following semester.

Sounds like a good deal? Keep reading.

Variety of Classes

In my previous job in political science, courses were either semester long undergrad lectures or semester long Ph.D. seminars. At Washington University most of us taught one large undergrad lecture, one smaller upper division undergrad class, and one either an undergrad seminar or Ph.D. class.

In business schools, there are a much larger variety of potential classes types, but the variety of topics you can teach is smaller is more limited. Does that make sense?  Let me explain.

Some business schools don’t have any undergraduate majors, while others (like GW) have a large undergraduate population. Ph.D. programs in business schools tend to be very small and many departments only offer one Ph.D. seminar across the whole department. For most business school faculty, you are teaching MBA students and most of these classes are the “core” classes. For me this would be teaching “global strategy” or “international management”.

But business schools have plenty of master’s classes that range from the regular daytime MBA, evening MBA, online MBA, accelerated MBA, executive MBA as well as numerous specialized masters programs. These programs come in different forms, ranging from a regular 15 week course, 7 week courses, to anything else you can imagine.  But these are different forms of similar core courses.

I found selecting my courses to be especially daunting given this variety of potential courses, students, and schedules. In my two years at GW I taught one 7 week MBA course, an large introduction to globalization (200+ students), an advanced undergrad course (50 students) and a Ph.D. course. If I stuck around there was a one week DC focused course, consulting abroad, as well as some weekend courses I would consider.

Course Choice

Unlike many social science departments, professional schools have much more rigid curricula and thus provide a much smaller number of courses. Similar to law schools that require every student to take courses such as Contracts and Civil Procedure, business schools have a large number of required courses that require staffing. This means that many faculty have very few choices on which courses they will offer, and in many cases faculty are hired  to teach a specific course. We currently have a position at GW that is essentially to help teach one of our core MBA classes (Global Perspectives).

Thus while there are many different course formats, most of these are variants of core courses and not electives.

On-Load vs Off-Load

What I found especially striking in the business school was the constant need for additional courses. Faculty were expected to fulfill their teaching obligations (three courses a year for research active faculty) and often had the opportunity to teach additional courses “off load” for additional pay. For many business schools these additional courses pay an additional 1/12 of your base salary for a 3 credit course.

I was under the impression that many business school faculty engage in consulting for additional income. After joining the business school it is clear that additional teaching, including more specialized certificate programs and executive education is that way that faculty can earn an extra income and provide a service to the school.  But this comes at a cost to your research time.

Evaluation and Teaching Quality

Before moving to my new job in a business school I asked a few other faculty that made a similar transitions about their experience. Despite what sounded like a comparable teaching experience, most of the faculty reported that they spent more time prepping courses and more overall energy in teaching at a business school.

I came from a department with great teachers (political science at WashU has some fantastic teachers who are even better researchers) and I generally had very strong course evaluations. I thought this tough transition wouldn’t apply to me.

Two years later I find that I am still spending considerably more time and energy teaching than I did in my previous job. Why?

First, there is considerably more pressure to be a “good teacher” in a business school. Like I said, my previous job had lots of great teachers, but in my current job we review other syllabi, talk about course evaluations, and have lots of opportunities to receive additional training as a teacher. Some of the teaching opportunities, such as executive education, are only open to some of the best teachers.

Much of this is based on teaching evaluations. I am very mixed on using this criteria for evaluating teaching given the many biases in course evaluations and the weak correlation between ratings and learning. But my point is factual. Evaluations matter.

Second, and more importantly, I find that teaching in a business school is less closely engaged with my research than in political science. When teaching political science courses I assign academic research on the topic, discuss research design, data quality, and the link between theory and evidence.

We do some of this in my management courses, but there is more focus on practical decisions by managers and more focus on cases. I read dozens of cases before I assign a single case in my classes.  It is fun to learn new materials and it is exciting teaching, but it has very little synergy with the research part of my job.

This isn’t meant as a complaint.  Teaching is a larger part of your job in a business school and the goal conflict between teaching and research is much starker than my own experience in political science.

Thinking about a job in a business school?

Lots of students and colleagues have asked me about jobs in business schools. Do I like it? Would your recommend it to others?

This is a complicated question. It’s a very different world that consists of different criteria for research quality, a very different service and administrative role in the job, and substantially different teaching.

The focus of this blog post is only on teaching.  For non-academics this may not make sense.  The public perception of professor is solely as a teacher.  But for the academics reading this blog we all know the very brutal job market and high tenure standards make research an extremely important part of our jobs. And service to your department, university, and professional become more important the longer you are in the job.

In my opinion, teaching in a business school comprises a larger percentage of your job (as does school service) than a position in political science. This leaves less time for research (or sleep).

But the teaching definitely has its rewards and good teaching leads to additional professional opportunities. I was lucky at GW to teach undergrads, MBAs, and Ph.D. students. I really love teaching undergrad students, especially when much of the teaching is focused on a liberal arts education that has a learning for the sake of learning component. Ph.D. classes are also extremely engaging, where there is such a close relationship between teaching and research.

I also found that my limited exposure to MBA teaching was very positive as well. I love living in the world of ideas. But many of us academics have taken to blogging or writing op-ed pieces as a way to start engaging in policy debates. MBA teaching engages you in a different form of policy debate. You have the opportunity to help educate managers in their decisions. Some of these managers are in the public sector (especially in DC) but many of them are in private firms.  If you want to make a real world impact, the MBA classroom could be for you.

But this exciting opportunity comes at a cost. I found that my job in a business school tilted my time and energy much farther away from research than my years in political science. We all have different ideals on how we spend our professional time.  I guess I am voting with my feet.

I’m moving to Austin!

Ok, my whole family is moving. But this blog is my fault. I will leave them out of this.

I will be joining the University of Texas-Austin Department of Government in the Fall of 2016. I am moving to a new job and back to my old field of political science.  I’m very excited about the move, but I will definitely miss lots of people here at GW and in DC.

I’m planning a bunch of future blog posts reflecting on the differences between teaching, research, and job searches in business schools relatively to political science. But that is for the future.

But today I would like to thank the faculty, staff and students at George Washington University.

Happy Holidays!

Nate

What’s the matter with New Jersey?: Overpaying and not paying for economic development

I’ve started yet another project on the costs of offering subsidies to individual firms. These economic development incentives are especially common in the United States where states and cities often compete with each other over firms.

One part of the project is compiling incentives data contained in a database called Incentive Monitor, which is one of the most comprehensive databases of incentives. There are other incentive databases, including Good Jobs First Subsidy Tracker.

Using the Incentive Monitor database we can get a snapshot of the use of incentives in the United States. In the year 2014 alone I can identify over 2,000 individual economic development incentives at an aggregate cost of $10 billion, although this largely ignores many of the local economic development incentives such as tax incremental financing. The incentives in this data base are credited with creating a total of just under 309,000 jobs, or just over $32,000 per job.

These aggregate numbers can be misleading, where the over 2,000 economic development programs are each associated with fiscal costs and job creation numbers. In some cases these incentives are associated with projects that create zero jobs, either these are associated with other goals, such as promoting renewable energy, or in “safeguarding” jobs from being poached by other cities, states, or countries.

By removing the incentives that have no state job creation numbers, the average amount of incentives per project remains at over $31,500. Whether you think this is a big or small number is up to you. What I found interesting is the dramatic differences in the cost of incentives per job by state.

Some states provide either provide no incentives or the number of observations are so small we don’t want to draw inferences. But for the states that offered at least ten incentives in 2014, there is a clear pattern.

First, a number of states provide incentives, but at a substantially lower cost per job than the national average. A large cluster of states offer incentives at cost per job in the teens. What do states like Florida ($10,000 per job), Indiana ($12,000), Maryland ($18.521) and Mississippi ($19,136) have in common?

The Pew Charitable Trusts has been active in helping states reform their economic development programs. They have identified states that have reformed their programs from 2012 to 2014, and five of these states are in our data set. Which states reformed their incentive programs? Florida, Indiana, Maryland, and Mississippi along with Louisiana. We’ll get to Louisiana in soon.

These states that have enacted reforms aren’t the only states with less costly incentive programs, and this description doesn’t make a causal claim that these reforms have reduced the costs of these programs. But there is clearly a cluster of reformer states that have much less costly incentive programs.

Second there are a number of states such as Connecticut that were consistently above the average. Investors in Connecticut were offer 87 incentives at an average cost per job of $59,000. This average is driven up by two massive incentives to United Technologies and Praxair. But of the 87 incentives, 52 were above the national average in terms of costs per job.

Louisiana, one of the states that enacted reforms of their incentive programs, was very active in offering incentives (125 incentives in 2014) and at a cost of $58,083 per job. Iowa performed better but was still above the national average (58 incentives at a cost of $42,000 per job). Illinois provided 31 incentives at a cost of $63,000 per job.

Making relative comparison of these programs is difficult given the different goals of the programs and the additional incentive programs in states that may not be captured in this data. For example, the many tax increment financing districts in Illinois are not included in this data, which could potentially increase or decrease the cost per job averages.

The point is that there are three clusters. Low cost per job, high cost per job and New Jersey.

New Jersey offered 86 incentives at an average cost of over $238,500 per job. That can’t be correct. Let me double check.

It is correct. New Jersey has clearly offered some massive incentives that created very few jobs. For example, the highly criticized $82 million incentive to attract the Philadelphia 76ers practice facility is credited with 250 jobs ($328,000 per job). This helps us get to the $238,500 per job estimate.

But offering incentives at a whopping per job cost isn’t an outlier in New Jersey. It seems to be their economic development strategy.

If we rank all 2,000 plus incentives on a cost per job basis, six of the most expensive (per job) incentives are in New Jersey. This includes the high profile Subaru incentive to attract the company to Camden, New Jersey. Of the top 50 most expensive incentives per job deals, here is the state distribution.

1-New Jersey

Nineteen of the fifty most expensive incentives were New Jersey incentives.

What is happening in New Jersey?

Interestingly, economic development incentives are under fire in New Jersey, but probably not for the reason that you have expected. The state has under funded their economic development budget and the state owns companies their incentive checks. $785 million in checks.

The main reason is that these incentives, unlike many of the tax credits in other states, require a budget allocation. For example, in most other states an economic development agency can be authorized by the legislature to allocate tax credits, lowering the firm’s future tax bill.

In New Jersey, firms pay their taxes, but are issued a rebate from the state coffers. What is the difference? The difference is that the state needs to have the money in an economic development fund to provide the rebate to firms. But with the budget crisis in New Jersey these programs have been underfunded and thus the rebate checks haven’t been sent out.

I wish I had some clever conclusions on what is happening in New Jersey. Something very, very different is going on in New Jersey. And that isn’t a compliment.

FOIA isn’t free

As part of a research project I submitted a number of freedom of information act (FOIA) requests. Most of my requests are at the state level, so different state sunshine laws shape what is required by agencies.

In the past I had a very mixed experience with these requests. Requests to state economic development agencies ranged from detailed data within a month to over a year wait to receive a simple spreadsheet.

My strategy was to send these agencies a polite request for data and a description of my research project. It is hard to draw inferences on this limited FOIA experience, so I hunted around on work on FOIA requests and learned about bit about FOIA requests.

Learning from the FOIA Requests of Others

Quite a few NGOs provide detailed advice on FOIA requests. Many of these agencies coach requesters to make very formal requests that provides details on state FOIA laws, know what exact data you want, and reference the legally required respond time. Walk in like you own the place is the saying that I have heard many times.

A number of other transparency groups provide similar recommendations and some of keep tabs on backlogs in FOIA requests. Public Citizen has a nice guide and a sample FOIA letter.  Here are resources for investigative reporters.

A nice template for those interested in state economic development is from Good Jobs First.

One issue with FOIA requests isn’t only the wait time, but the potential costs. See the above guides where journalists can have fees waved. But for academics there seems to be more discretion from agencies. My own FOIA requests have ranged from free to a quote of $1,000.

Learning from the FOIA Responses of Agencies

FOIA requests are generally made to an agency and whatever information they provide as part of the request goes directly to the requestor. But in other cases, agencies such as the EPA make FOIA data public. So if you request data the agency will send you the data and publicly post the request and data.

One organization, MuckRock directly posts FOIA requests, correspondence with the agency, and the final data on their website. Thus even for agencies that don’t make this request public, MuckRock posts it for everyone to see.

This can be extremely useful for academic research. For example, I examined FOIA requests on issues related to both trade and economic development subsidies.   This search not only uncovered some interesting FOIA request, there was also a FOIA request for all USTR FOIA requests.

What I found interesting in these USTR FOIA requests is that you observe lots of FOIA requests by political parties. A large number of USTR FOIA requests were from the Democratic Senatorial Campaign Committee. Then a FOIA request from the National Republican Senatorial Committee to get the FOIA requests and data of the Democratic Senate Committee. The Republicans want to know what the Democrats found.

I also accidentally stumbled onto a request for all rejected vanity license plates in Maryland. They didn’t have the list of rejected plates, but they do have a guide that lists “objectionable plates” that plates that would be automatically rejected. Don’t open this in front of your teenage kids or if you plan on having a productive afternoon. There are thousands of these plates.

There’s many “failed requests”. Many agencies don’t respond, require the requester to be a resident of the state, charge large amounts of money, or claim that they are exempt from the request.

The Opposition to FOIA Requests

When looking at these FOIA requests I notice that MuckRock allows organizations (including investigative journalists) to have their data embargoed to allow time for them to write up their “scoop”.

I honestly hadn’t thought about this, but it seemed obvious to me that even journalists have mixed views about FOIA requests. Having access to data that nobody else has provides a competitive advantage over other journalists.

It turns out that new Federal Government FOIA policy have caused a stir. Many agencies are now required to make their FOIA requests and data public.

This has lead to a backlash from at least some journalists.  The worry is that journalists using FOIA requests for a story will be making their request public for others (including other journalists) to see.

As a professor in a business school these concerns about competitive advantages make perfect sense. Investigative journals want a scoop, and collecting information is only a small part of the process. Journalists could feel pressure to quickly write up stories or simply stop investing time and money into FOIA requests.

But there is at least some irony that intrepid investigative journalists pushing for transparency of government wanting to be the gatekeepers of this transparency.

UPDATE

I just received data from a FOIA request for Virginia economic development data.  The FOIA representative from YesVirginia contacted me a couple days after my submission clarifying my request.  And then delivered almost everything I wanted just a few days later.  Kudos to the staff at Yes Virginia.

Data Access and Replication Policies in Management

There’s a heated exchange in political science over a new Data Access and Replication Transparency (DA-RT) initiative.

Twenty-seven political science journals have signed onto a data share policy. Opposition to this policy has organized a signature campaign aimed at delaying implementation of DA-RT that now includes over 1,000 signatures.

For a list of the many blog posts on the topic can be found here.

My contribution is to give a quick overview of data access and replication practices from management. Below I will provide an overview of the data replication policies of these journals.

If you don’t get to the actual description I can give a quick summary that will make nobody happy.

If we divided the journals between mostly quantitative and qualitative journals, I think we would find that all of the journals with mandatory data access and replication policies largely focus on quantitative analysis.

But many of the journals with quantitative orientations, including a top statistics journal, don’t have a mandatory replication policy. The three big financial journals attempted a replication policy and it failed due to a major backlash from the editorial board.

Replication policies aren’t easy for any journal. But there seems to some models out there that can help guide us. But in this summary I don’t see any obvious model from a journal that focuses on qualitative work.

If you are still with me, I can provide some details.

Update: Here are replication studies in political science and economics.

Business School Publishing

Business schools are eclectic mixes of faculty with background in different disciplines. My department (International Business) has Ph.D.s in economics, management, political science, psychology, and sociology. This leads to complications in how to weight different types of research contributions (books, cases, journal article, policy pieces, etc). I recently posted that very few political science journals “count” in many business schools.

The narrowest list of the top business journal is the Financial Times 45 (FT 45). This is the gold standard of top business journals and I don’t know of a single department that wouldn’t count these journals as top contributions.

I went through the FT 45 journals in search of formal replication and data access policies. Of the 45 journals the vast majority of them have no stated data access or replication policy. At the end of this post I include the full list of FT 45 journals and their policies. If I missed something, please let me know.

Journals with Mandatory Data Access Policy

Many of the journals in the FT 45 with formal policies are economics and psychology journals. The American Economic Review, for example, has a clear mandatory data availability policy and has a repository for randomized control trails. Econometrical and the Journal of Political Economy have similar policies.

The Journal of Applied Psychology, the Journal of Consumer Research and the Journal of Consumer Psychology, and Marketing Science have mandatory data access policies. The Journal of Consumer Psychology also requires IRB verification for human subjects research.

This is the full list of FT 45 journals with mandatory data access and replication policies.  That’s it.

Journals that Encourage Data Access

There are a handful of journals that urge authors to provide their data or at least provide extensive documentation of their data and analysis.

For example, American Accounting Review has the following statement:

The AAA’s Executive Committee policy (originally adopted in 1989, and amended in 2009) is that the objective of the Association-wide journals (The Accounting Review, Accounting Horizons, Issues in Accounting Education) is to provide the widest possible dissemination of knowledge based on systematic scholarly inquiries into accounting as a field of professional research and educational activity. To fulfill this objective, authors are encouraged to make their data available for use by others in extending or replicating results reported in their articles. Please see the policy on Data Integrity adopted in March of 2015 for more information.

The Journal of Accounting Research has one of the more complicated policies that fall short of full replication. The full description of their policy is can be found on their website.  One interesting aspect of this policy include verification by the journal of access to the proprietary.  If you have top secret data you need to prove that you accessed this data. Second:

Prior to final acceptance of the paper, the computer program used to convert the raw data into the dataset used in the analysis plus a brief description that enables other researchers to use this program. Instead of the program, researchers can provide a detailed step-by-step description that enables other researchers to arrive at the same dataset used in the analysis. The purpose of this requirement is to facilitate replication and to help other researchers understand in detail how the sample was formed, including the treatment of outliers, Winsorization, truncation, etc. This programming is in most circumstances not proprietary. However, we recognize that some parts of the data generation process may indeed be proprietary or otherwise cannot be made publicly available. In such cases, the authors should inform the editors upon submission, so that the editors can consider an exemption from this requirement.

Given the extensive use of proprietary data in Accounting, my opinion is that this is close to a mandatory replication policy. Authors using proprietary data such as Compustat must prove they have access to this database and provide clear instructions on how others with access could extract the data.

The Journal of the American Statistical Association is a clear example of a journal that recommends data sharing but doesn’t make it mandatory.  This one was honestly a surprise to me.

Some publishers have general statements that apply to all of their journals.  This one is from from Elseiver:

Research data is the foundation on which scientific, technical and medical knowledge is built, but there are challenges in making it accessible and shareable. In line with the STM Brussels Declaration, Elsevier envisions a future in which data can be easily and effectively stored, shared, discovered and used, in support of researchers and for the advancement of science and health. However, there are challenges in making research data accessible and shareable. We have developed our research data policy to address these challenges and will continue to actively support further

Despite the lack of technology, the publisher does allow you to publish your data in their “Data-in-Brief” feature for a fee of $200.  Replication isn’t cheap.

The Journal of Applied Psychology (American Psychological Association) has the following policy:

In addition, APA Ethical Principles specify that “after research results are published, psychologists do not withhold the data on which their conclusions are based from other competent professionals who seek to verify the substantive claims through reanalysis and who intend to use such data only for that purpose, provided that the confidentiality of the participants can be protected and unless legal rights concerning proprietary data preclude their release” (Standard 8.14). APA expects authors to adhere to these standards. Specifically, APA expects authors to have their data available throughout the editorial review process and for at least 5 years after the date of publication.

Journals with No Data Access or Replication Policies

In total, 35 of the FT 45 either had no data access policy or explicitly stated that data sharing isn’t mandatory.

In my review of management journal websites I couldn’t find any explicit journals statements with data access or replication policies. For example, the three Academy of Management (AOM) journals make no mention of data access or replication.

The Strategic Management Journal has no formal replication policy, although they do have a statement about P-hacking:

Data Snooping and P-hacking
SMJ strongly disapproves of data snooping and p-hacking practices in empirical research. Authors of submitted papers should not search databases for statistically significant coefficients with the intention of subsequently formulating hypotheses that fit the significant coefficients. Authors also should not adapt experimental designs with the primary intention of producing statistically significant results. In addition, authors of submitted papers should address the material significance (magnitude) of the results, in addition to statistical significance.

Finance journals also lacks formal policies. According to a previous editor, the Journal of Finance proposed a replication policy (along with the Journal of Financial Economics and the Review of Financial Studies). This caused a major backlash and was shelved.

Two economics journals are also silent on data access.

Summary of Journal Practices

  • In-house replication of articles: None of the FT45 provide an in-house replication of journals. In contrast, numerous political science journals such as the American Journal of Political Science performs a pure replication.
  • Mandatory data sharing: Seven journals have mandatory data sharing policies. These journals are economics journals, psychology journals, or marketing journals. The American Economic Review is a good model for this type of data sharing.
  • Sharing code for propriety data: The Journal of Accounting Research provides details on expectations for the extraction of proprietary data.
  • Encouraging Data Sharing: Some journals such as the Journal of the American Statistical Association provide editorial statements encouraging
  • No Data Sharing Policies: Essentially all of management has no policy. 

What do make of this evidence?

This isn’t my job. It is up to you to draw your own conclusions.

Here is a quick summary of the FT 45.

FT 45 Data Access and Replication Policies

Academy of Management Journal (Academy of Management)

No stated data access or replication policy

Academy of Management Perspectives (Academy of Management)

No stated data access or replication policy

Academy of Management Review (Academy of Management)

No stated data access or replication policy

Accounting, Organisations and Society (Elsevier)

No stated data access or replication policy

The Accounting Review (American Accounting Association)

No mandatory data access or replication policy but detailed statement from AAA encourage data sharing and have detailed guidelines.

Administrative Science Quarterly (Cornell University)

No stated data access or replication policy

American Economic Review (American Economic Association)

Mandatory data availability policy.

California Management Review (UC Berkeley)

No stated data access or replication policy

Contemporary Accounting Research (Wiley)

No stated data access or replication policy

Requires copies of other papers using this same data.

Econometrica (Econometric Society, Wiley)

Mandatory replication policy

Entrepreneurship Theory and Practice (Baylor University, Wiley)

No stated data access or replication policy

Harvard Business Review (Harvard Business School Publishing)

No stated data access or replication policy

Human Resource Management (Wiley)

No stated data access or replication policy

Information Systems Research (Informs)

No stated data access or replication policy

Ethical guidelines only documenting process:

Journal of Accounting and Economics (Elsevier)

No stated data access or replication policy

Journal of Accounting Research (University of Chicago, Wiley)

No blanket mandatory data sharing policy. But there confirmation of proprietary data and requires details on data collection.

Journal of Applied Psychology (American Psychological Association)

Formal data access policy

Journal of Business Ethics (Kluwer Academic)

No stated data access or replication policy

Journal of Business Venturing (Elsevier)

No stated data access or replication policy

Journal of Consumer Psychology (Elsevier)

Mandatory data access policy based on APA

 

Journal of Consumer Research (University of Chicago)

Data statement paragraph and mandatory data sharing.

 

Journal of Finance (Wiley)

No stated data access or replication policy

A letter from the previous editor on the backlash shelving the policy

Journal of Financial and Quantitative Analysis (Cambridge University Press)

No stated data access or replication policy

Journal of Financial Economics (Elsevier)

No stated data access or replication policy

Journal of International Business Studies (Academy of International Business)

No stated data access or replication policy

Journal of Management Studies (Wiley)

No stated data access or replication policy

Journal of Marketing (American Marketing Association)

Data access not required

Journal of Marketing Research (American Marketing Association)

Data access not required

 

Journal of Operations Management (Elsevier)

No stated data access or replication policy

Journal of Political Economy (University of Chicago)

Mandatory replication policy

 

Journal of the American Statistical Association (American Statistical Association)

Optional Data Policy

Management Science (Informs)

No stated data access or replication policy

Marketing Science (Informs)

Mandatory replication policy

MIS Quarterly (Management Information Systems Research Centre, University of Minnesota)

No stated data access or replication policy

Operations Research (Informs)

No stated data access or replication policy

Organization Science (Informs)

No stated data access or replication policy

Organization Studies (SAGE)

No stated data access or replication policy

Organizational Behaviour and Human Decision Processes (Elsevier)

No stated data access or replication policy

Production and Operations Management (Wiley)

No stated data access or replication policy

Quarterly Journal of Economics (MIT)

No stated data access or replication policy

Rand Journal of Economics (The Rand Corporation, Wiley)

No stated data access or replication policy

Review of Accounting Studies (Springer)

No stated data access or replication policy

Review of Financial Studies (Oxford University Press)

No stated data access or replication policy

Sloan Management Review (MIT)

No stated data access or replication policy

Strategic Management Journal (Wiley)

No stated data access or replication policy

Learning to Replicate

Right now the internet is blowing up with more news on the retracted LaCour and Green study.

But this post is mostly about a conference I attended today on replication in the social sciences hosted by the International Institute for Impact Evaluation (3ie). The conference followed Chatham House rules, essentially asking us to not attribute any of the comments to individuals.

The conference was mostly focused on development economics. 3ie has an ambitious replication initiative focused on a handful of influential papers. What I found really interesting is the tone of the discussion in this audience versus most of my interactions with political scientists. Numerous commentators make direct or indirect claims that the editors of the (econ) journals were the problem. They either had no replication policy or didn’t enforce their replication standards. One overview of the status of replication in economics is here.

As a political scientist, I felt like I was going back in time. I haven’t checked every journal, but I was under the impression that just about every major political science journal has a replication policy in place.

Now I checked. There is a damning study claiming only 18 of 180 political science journals have replication policies. The good news is that the American Political Science Review, American Journal of Political Science, and Journal of Politics all have replication policies and that impact factor is one of the best predictors of a replication policy.  But a few excellent journals do not.

Many journals now require archived data as a condition for publication and a few journals like Political Science Research and Methods and the American Journal of Political Science require a technical replication before publishing. My prediction is that within the next five year most of the major political science journals will require replication as a condition for publication.

I don’t have too much more to say about this, other than there seems to be major cultural differences in views on replication across fields that affects data sharing.

A number of presenters discussed the different types (and terms) of replication. Some of this seemed like inside baseball until we began discussing Michael Clemens paper on defining replication. I think the most important idea here is that the term “replication” carries a lot of baggage for the original researcher. Individuals (and journals) should encourage “replication” but be very careful in labeling a study as “failing to replicate”. For example, if a scholar made up the data, that is clearly a failure to replicate. But what if a study of land titling in country X doesn’t yield the same results as an identical land titling study as country Y. Labeling the original study as a failure to replicate suggests wrongdoing by the original author.

For those of us that have followed some of the nasty exchanges between original authors and authors of replication studies (some of these 3ie studies) it is easy to see how the threat of a label of “failed replication” could lead to defensiveness.  Many of the discussants highlighted the importance of engaging the original authors in a dialogue.  Our goal shouldn’t be to catch mistakes by authors. It is to correct them to help us learn more about the world.

One of the most thought provoking presentation was by Brian Nosek on the work being done through the Open Science Framework. Brian presented a number of replication studies.  This includes:

  • The Reproducibility Project: Using a single year (2008) and three top psychology journals, 270 authors attempted to replicate 100 studies. There is a lot of information here, but the quick overview is that a very large percentage of the studies that attempted to collected new data and conduct the same analysis had insignificant results and smaller effects sizes. Given Clemens point on the definition of replication, let’s not call these failed replications. At least one person in the room labeled this publication bias. Either way, it is problematic.
  • The Many Labs Project assigned 27 teams to attempt to answer the same substantive questions with the same data, but they had the discretion on the coding, method, covariates and any other specification decisions. There was huge variation in the answers that different findings.
  • Nosek presented some evidence on which types of studies failed to replicate. He went through this very quickly and I can’t find any supporting materials online. But what was especially interesting is that an elite survey of experts in the field didn’t do an especially great job in predicting which studies replicated, but a prediction market did. The prediction markets under-predicted failed replications, but overall did a pretty decent job.  Basically, given the right (monetary incentives), experts could sniff out the studies that weren’t going to replicate.

Within all of the presentations there was an acknowledgement of incentive problems from original authors (to have a finding and not be proven wrong) and replication authors (to find something new/wrong to report). But I didn’t hear any clear solutions.

Finally, there was very little discussion of the LaCour and Green controversy. One commentator noted that this is a case of  replication success, not a failure. By making the data available science/Science was able to correct itself.

I’m not sure I am as optimistic. But it was nice to see that our discussions weren’t completely derailed by this crazy situation.

A few more sources.

Great resources from Gary King on replication here

Another study that attempts to define replication here

I forgot about Andrew Gelman’s Garden of Forking Paths

Which political science journals count?

I am finishing my first academic year as faculty in a business school after twelve years in a political science department. As the only political scientists in our school I often have to explain the different political science journals.

Business schools are an exciting mix of faculty with Ph.D.s in management, economics, sociology, history, statistics, psychology, and political science. On top of these differences in training, faculty are housed in departments as different as marketing, finance, accounting, strategy, international business, decision science, and information systems.

These differences often require benchmark “lists” of accepted journals for everything from internal promotion, allocating teaching loads (reduced teaching for “research active” faculty) to the ability to advise Ph.D. students.

There are three broadly used lists. The FT 45, the UT-Dallas list, and the most comprehensive Association of Business Schools (ABS). The 2015 ABS guide just came out.  ABS ranks over 1,400 journals from 4-1 with 4 being the top ranking. Ok, they also give a * to some 4 journals, giving them. Grade inflation.

Just to give you some flavor for this, ABS ranks many economics journals. 23 are ranked 4 and 68 are ranking three.

Psychology also has their own set of rankings . Other fields, like sociology are ranked within the catch-all “Social Sciences” lists.

Without editorializing, here is the full list of all of the political science journals I could find.

Political Science Journals Ranked “4”

 

None

Political Science Journals Ranked “3”

Quarterly Journal of Political Science (added in 2014)

Electoral Studies (added in 2014)

Public Choice

New Political Economy

Public Opinion Quarterly (added in 2014)

Review of International Political Economy

West European Politics (added in 2014)

Political Science Journals Ranked “2”

Economics and Politics

Political Studies

Political Science Journals Ranked “1”

None

Family transcript

This blog is mostly about work, but I can’t help but occasionally post about my family.  I am a single dad this weekend with my 1.5 years old (Stanley) and 3.5 year old (Walter) boys. Here is a transcript of our latest conversation.

Walter: Turtles come from eggs. And tadpoles are frogs.
Nate: Right. Where did you learn this? This is great.
Stan: Yeah.
Walter: And frogs jump really high.
Stan: Yeah.
Nate: You’re right!
Stan: Yeah.
Walter: Frogs can jump over rainbows.
Stan: Yeah.
Nate: I don’t think that is true.
Walter: When little boys jump over rainbows they hurt their heads. But frogs never hurt their heads.
Stan: Yeah.
Nate: I think frogs can hurt their heads. Stan, stop telling him yes.
Stan: Yeah.
Walter: No, frogs never hurt their heads. Little boys can hurt their heads. But only when they jump over rainbows.
Stan: Yeah.
Nate: I give up.
Stan: Pancake.