The Scientific Method in Practice: Reproducibility in the Computational Sciences

  title={The Scientific Method in Practice: Reproducibility in the Computational Sciences},
  author={Victoria Stodden},
  • V. Stodden
  • Published 9 February 2010
  • Computer Science
Since the 1660’s the scientific method has included reproducibility as a mainstay in its effort to root error from scientific discovery. [] Key Result It provides evidence in the debate about whether scientists’ research revealing behavior is wholly governed by considerations of personal impact or whether the reasoning behind the revealing decision involves larger scientific ideals, and secondly, this research describes the actual sharing behavior in the Machine Learning community.

Research method in AI

A survey design for evaluating documentation of published papers, an evaluation of two leading AI conference series, and suggested incentives to facilitate the reproducibility of AI research are suggested.

Using ReproZip for Reproducibility and Library Services

It is believed that libraries and archives can leverage ReproZip to deliver more robust reproducibility services, repository services, as well as enhanced discoverability and preservation of research materials, applications, software, and computational environments.

R Markdown

Training students in R Markdown will introduce to the workforce a new class of data analysts with an ingrained, foundational inclination toward reproducible research.

A Model-Driven Engineering Framework for Computational Replicability and Reproducibility

It is shown how the process has been used to successfully produce a Platform Independent Model from the essential components of a model developed in a Platform specific environment and the details of how this Platform Independent representation can be used to generate a second model in the original platform are confirmed.

Performing reproductions to understand the state of reproducibility in current AI research

This project attempts to provide a better understanding of the state of reproducibility in the field, and identifies issues limiting reproductions in AI from recent years.

A Minimum Standard for Publishing Computational Results in the Weather and Climate Sciences

A procedure for reporting computational results that was employed in a recent Journal of Climate paper is described, developed to be consistent with recommended computational best practices and to address the reproducibility crisis.


In this commentary, the New England Journal of Medicine published a commentary calling scientists who repurpose data "research parasites" who misuse data generated by others to demonstrate alternative hypotheses.

Towards standard practices for sharing computer code and programs in neuroscience

This article lists practical suggestions to maximise the reproducibility of authors' work, and how best to maximize the chances that other scientists can reproduce the authors' computations or reuse their methods on their data.

Computational reproducibility in geoscientific papers: Insights from a series of studies with geoscientists and a reproduction study

Advice is proposed for authors to overcome the issues around reproducibility in the computational geosciences using the R code.

Schroedinger’s Code: A Preliminary Study on Research Source Code Availability and Link Persistence in Astrophysics

Examining software usage in a sample set of astrophysics research articles published in 2015 and searching for the source codes for the software mentioned in these research papers found that 285 unique codes were used, 58% of which offered the source code for download.



15 Years of Reproducible Research in Computational Harmonic Analysis

The approach to reproducible computational research is reviewed, how the approach has spread over time, and some of the arguments for and against working reproducibly are discussed.

Enabling Reproducible Research: Open Licensing for Scientific Innovation

The Reproducible Research Standard is proposed both to ensure attribution and facilitate the sharing of scientific works to encourage reproducible scientific investigation, facilitate greater collaboration, and promote engagement of the larger community in scientific learning and discovery.

How Open Source Software Works: 'Free' User-to-User Assistance?

It is found that the Apache field support system functions effectively and that, when the help system is partitioned into its component tasks, 98% of the effort expended by information providers in fact returns direct learning benefits to those providers.

How Much Science Can You Have at Your Fingertips?

  • I. Good
  • Education
    IBM J. Res. Dev.
  • 1958
Numerous suggestions are presented for helping men to learn and retrieve scientific information, with or without artificial aids, and a warning that some of the world's greatest scientists have done their best work when they were given plenty of time to think their own thoughts.

The Structure of Scientific Revolutions

A good book may have the power to change the way we see the world, but a great book actually becomes part of our daily consciousness, pervading our thinking to the point that we take it for granted,

The Sociology of Science: Theoretical and Empirical Investigations

"The exploration of the social conditions that facilitate or retard the search for scientific knowledge has been the major theme of Robert K. Merton's work for forty years. This collection of papers

Do Open-Access Articles Have a Greater Research Impact?

Although many authors believe that their work has a greater research impact if it is freely available, studies to demonstrate that impact are few. This study looks at articles in four disciplines at

The Golem: What Everyone Should Know about Science

The Mathematical Association of America, 1993 372 pages $35.00 "A Cat That Can't Be Caught" is the first chapter of this quite readable story of the life of Eric Temple Bell. Constance Reid carries

Breakdown Point of Model Selection When the Number of Variables Exceeds the Number of Observations

  • D. DonohoV. Stodden
  • Mathematics
    The 2006 IEEE International Joint Conference on Neural Network Proceedings
  • 2006
This work points out that when p > n, there is a breakdown point for standard model selection schemes, such that model selection only works well below a certain critical complexity level depending on n/p, and applies this notion to some model selection algorithms (Forward Stepwise, LASSO, LARS) in the case where pGtn.

Science of Science and Reflexivity

Over the last four decades, the French sociologist Pierre Bourdieu produced one of the most imaginative and subtle bodies of social theory of the postwar era. When he died in 2002, he was considered