Replication in economics

From ReplicationWiki
(Redirected from Replication in Economics)
Jump to: navigation, search

“Research that is not replicable is not science and cannot be trusted either as a part of the profession’s accumulated body of knowledge or as a basis for policy.” [1]

The general aim of replication is to review the results and to test their relevance. While in other sciences replicability is regarded as a fundamental principle for research and for the publication of results, in empirical economics applied econometric results are based on data and calculations that usually do not get published and that are not routinely controlled. For authors of empirical studies the workload needed to make their material replicable is not awarded in the same way as publishing new irreplicable studies. Neither is authoring replication studies, so there is no great danger of getting penalized for irreplicable research. Although there have already been warnings in the literature for decades about the dangers of the neglect of replication, in empirical economics it is still not treated as a top priority.[2] Only few journals publish replication studies and have introduced policies that should help to ensure the replicability of their published results. Even in cases of data policies not all articles are replicable.[3][4]


Incentive problem in economics

Various studies found a lack of incentives for replication and warned of the hindrance this poses to progress in economic thinking. On the one hand for researchers there are only few incentives to replicate papers and publish their results. On the other hand their must also be incentives for authors to spend time to compile and document every step of their research such that others can use it. The Journal of Applied Econometrics introduced a replication section in 2003, other journals also publish replications. This is a first step to increase the number of published replication studies, another step is to raise the importance attached to them in the scientific community. For replicablility it is important that journals provide clear guidelines and an archive where the material is stored. But archives alone are not sufficient, there must also be an incentive scheme to submit good material, as otherwise some authors only submit partial data, partial code, data that does not run with code, code fragments that do not run, etc.[5] For a fully replicable article it is important that all code and data are available including raw data and information about data cleaning. Developments in software can help researchers to improve replicability.[6]


Obviously, for empirical articles to be replicable they need to meet a certain degree of transparency. Accordingly, there has to be access to the data and code that the calculations are based on. In most cases further descriptions of data and code are necessary to be able to run the calculations. Therefore, it is common to publish read me-files as well. For empirical articles in economics B.D. McCullough proposes 10 conditions on data availability, program code, readme-file and journal publication policy to achieve replicability.[7] There are few attempts of scientific journals to enhance replicability by defining adequate publication policies, cf. Journal publication policies.

Positive effects of replication

  • Without replicability, mistakes and even fabrication are more difficult to discover
  • Help for further studies (scientific progress)
  • Students can learn from replication in econometrics classes

Types of replication

There are two general types of replication studies. Replication studies in a “narrow” and “wide” sense.[8] Hamermesh distinguishes between pure replication and scientific replication.[9]

Replication in a narrow sense

Narrow, or pure, replication means first checking the submitted data against the primary sources (when applicable) for consistency and accuracy. Second the tables and charts are replicated using the procedures described in the empirical article. The aim is to confirm the accuracy of published results given the data and analytical procedures that the authors write to have used.

Replication in a wide sense

Replication in a wide sense is to consider the empirical finding of the original paper by using either new data from other time periods or regions, or by using new methods, e.g., other specifications. Studies with major extensions, new data or new empirical methods are often called reproductions.

Possible replication results

The results of the replications are classified in four categories.


Results could be replicated without major deviations from the published results.

partially successful

Key results could be replicated but some deviations from published results.


Key results could not be replicated, significant deviations from the published results.

not replicable

Impossible to obtain the published results from the available material.


  1. McCullough, B.D., Hrishikesh D. Vinod (2003), “Verifying the Solution from a Nonlinear Solver: A Case Study”, American Economic Review, 93(3), 873-92.
  2. DeWald, William, Jerry Thursby, Richard Anderson (1986), “Replication in empirical economics: the Journal of Money, Credit and Banking project”, American Economic Review, 76 (4), 587-603.
  3. McCullough, B.D., Kerry Anne McGeary, Teresa D. Harrison (2006), “Lessons from the JMCB Archive“, Journal of Money Credit and Banking, 38(4), 1093-1107.
  4. Glandon, Philip (2010), Report on the American Economic Review Data Availability Compliance Project
  5. Anderson, Richard, William H. Greene, B.D. McCullough, H.D. Vinod (2008), “The role of data & program code archives in the future of economic research”, Journal of Economic Methodology, 15(1), 99-115.
  6. Koenker, Roger, & Achim Zeileis (2009), “On Reproducible Economics Research”, Journal of Applied Econometrics, 24(5): 833-847.
  7. McCullough, B.D. (2007), "Got Replicability? The Journal of Money Credit and Banking Archive", Econ Journal Watch, 4 (3), 326-37.
  8. Pesaran, H. (2003), "Introducing a replication section", Journal of Applied Econometrics, 18: 111.
  9. Hamermesh, Daniel S. (2007), “Viewpoint: Replication in Economics”, Canadian Journal of Economics, 40(3): 715-733.

External links

Replication Project in Experimental Psychology

Personal tools