Data Access and Replication Policies in Management

There’s a heated exchange in political science over a new Data Access and Replication Transparency (DA-RT) initiative.

Twenty-seven political science journals have signed onto a data share policy. Opposition to this policy has organized a signature campaign aimed at delaying implementation of DA-RT that now includes over 1,000 signatures.

For a list of the many blog posts on the topic can be found here.

My contribution is to give a quick overview of data access and replication practices from management. Below I will provide an overview of the data replication policies of these journals.

If you don’t get to the actual description I can give a quick summary that will make nobody happy.

If we divided the journals between mostly quantitative and qualitative journals, I think we would find that all of the journals with mandatory data access and replication policies largely focus on quantitative analysis.

But many of the journals with quantitative orientations, including a top statistics journal, don’t have a mandatory replication policy. The three big financial journals attempted a replication policy and it failed due to a major backlash from the editorial board.

Replication policies aren’t easy for any journal. But there seems to some models out there that can help guide us. But in this summary I don’t see any obvious model from a journal that focuses on qualitative work.

If you are still with me, I can provide some details.

Update: Here are replication studies in political science and economics.

Business School Publishing

Business schools are eclectic mixes of faculty with background in different disciplines. My department (International Business) has Ph.D.s in economics, management, political science, psychology, and sociology. This leads to complications in how to weight different types of research contributions (books, cases, journal article, policy pieces, etc). I recently posted that very few political science journals “count” in many business schools.

The narrowest list of the top business journal is the Financial Times 45 (FT 45). This is the gold standard of top business journals and I don’t know of a single department that wouldn’t count these journals as top contributions.

I went through the FT 45 journals in search of formal replication and data access policies. Of the 45 journals the vast majority of them have no stated data access or replication policy. At the end of this post I include the full list of FT 45 journals and their policies. If I missed something, please let me know.

Journals with Mandatory Data Access Policy

Many of the journals in the FT 45 with formal policies are economics and psychology journals. The American Economic Review, for example, has a clear mandatory data availability policy and has a repository for randomized control trails. Econometrical and the Journal of Political Economy have similar policies.

The Journal of Applied Psychology, the Journal of Consumer Research and the Journal of Consumer Psychology, and Marketing Science have mandatory data access policies. The Journal of Consumer Psychology also requires IRB verification for human subjects research.

This is the full list of FT 45 journals with mandatory data access and replication policies. That’s it.

Journals that Encourage Data Access

There are a handful of journals that urge authors to provide their data or at least provide extensive documentation of their data and analysis.

For example, American Accounting Review has the following statement:

The AAA’s Executive Committee policy (originally adopted in 1989, and amended in 2009) is that the objective of the Association-wide journals (The Accounting Review, Accounting Horizons, Issues in Accounting Education) is to provide the widest possible dissemination of knowledge based on systematic scholarly inquiries into accounting as a field of professional research and educational activity. To fulfill this objective, authors are encouraged to make their data available for use by others in extending or replicating results reported in their articles. Please see the policy on Data Integrity adopted in March of 2015 for more information.

The Journal of Accounting Research has one of the more complicated policies that fall short of full replication. The full description of their policy is can be found on their website. One interesting aspect of this policy include verification by the journal of access to the proprietary. If you have top secret data you need to prove that you accessed this data. Second:

Prior to final acceptance of the paper, the computer program used to convert the raw data into the dataset used in the analysis plus a brief description that enables other researchers to use this program. Instead of the program, researchers can provide a detailed step-by-step description that enables other researchers to arrive at the same dataset used in the analysis. The purpose of this requirement is to facilitate replication and to help other researchers understand in detail how the sample was formed, including the treatment of outliers, Winsorization, truncation, etc. This programming is in most circumstances not proprietary. However, we recognize that some parts of the data generation process may indeed be proprietary or otherwise cannot be made publicly available. In such cases, the authors should inform the editors upon submission, so that the editors can consider an exemption from this requirement.

Given the extensive use of proprietary data in Accounting, my opinion is that this is close to a mandatory replication policy. Authors using proprietary data such as Compustat must prove they have access to this database and provide clear instructions on how others with access could extract the data.

The Journal of the American Statistical Association is a clear example of a journal that recommends data sharing but doesn’t make it mandatory. This one was honestly a surprise to me.

Some publishers have general statements that apply to all of their journals. This one is from from Elseiver:

Research data is the foundation on which scientific, technical and medical knowledge is built, but there are challenges in making it accessible and shareable. In line with the STM Brussels Declaration, Elsevier envisions a future in which data can be easily and effectively stored, shared, discovered and used, in support of researchers and for the advancement of science and health. However, there are challenges in making research data accessible and shareable. We have developed our research data policy to address these challenges and will continue to actively support further

Despite the lack of technology, the publisher does allow you to publish your data in their “Data-in-Brief” feature for a fee of $200. Replication isn’t cheap.

The Journal of Applied Psychology (American Psychological Association) has the following policy:

In addition, APA Ethical Principles specify that “after research results are published, psychologists do not withhold the data on which their conclusions are based from other competent professionals who seek to verify the substantive claims through reanalysis and who intend to use such data only for that purpose, provided that the confidentiality of the participants can be protected and unless legal rights concerning proprietary data preclude their release” (Standard 8.14). APA expects authors to adhere to these standards. Specifically, APA expects authors to have their data available throughout the editorial review process and for at least 5 years after the date of publication.

Journals with No Data Access or Replication Policies

In total, 35 of the FT 45 either had no data access policy or explicitly stated that data sharing isn’t mandatory.

In my review of management journal websites I couldn’t find any explicit journals statements with data access or replication policies. For example, the three Academy of Management (AOM) journals make no mention of data access or replication.

The Strategic Management Journal has no formal replication policy, although they do have a statement about P-hacking:

Data Snooping and P-hacking
SMJ strongly disapproves of data snooping and p-hacking practices in empirical research. Authors of submitted papers should not search databases for statistically significant coefficients with the intention of subsequently formulating hypotheses that fit the significant coefficients. Authors also should not adapt experimental designs with the primary intention of producing statistically significant results. In addition, authors of submitted papers should address the material significance (magnitude) of the results, in addition to statistical significance.

Finance journals also lacks formal policies. According to a previous editor, the Journal of Finance proposed a replication policy (along with the Journal of Financial Economics and the Review of Financial Studies). This caused a major backlash and was shelved.

Two economics journals are also silent on data access.

Summary of Journal Practices

In-house replication of articles: None of the FT45 provide an in-house replication of journals. In contrast, numerous political science journals such as the American Journal of Political Science performs a pure replication.
Mandatory data sharing: Seven journals have mandatory data sharing policies. These journals are economics journals, psychology journals, or marketing journals. The American Economic Review is a good model for this type of data sharing.
Sharing code for propriety data: The Journal of Accounting Research provides details on expectations for the extraction of proprietary data.
Encouraging Data Sharing: Some journals such as the Journal of the American Statistical Association provide editorial statements encouraging
No Data Sharing Policies: Essentially all of management has no policy.

What do make of this evidence?

This isn’t my job. It is up to you to draw your own conclusions.

Here is a quick summary of the FT 45.

FT 45 Data Access and Replication Policies

Academy of Management Journal (Academy of Management)