Enhancing reproducibility for computational methods

@article{Stodden2016EnhancingRF,
  title={Enhancing reproducibility for computational methods},
  author={Victoria Stodden and Marcia K. McNutt and David H. Bailey and Ewa Deelman and Yolanda Gil and Brooks Hanson and M. Heroux and John P. A. Ioannidis and Michela Taufer},
  journal={Science},
  year={2016},
  volume={354},
  pages={1240 - 1241}
}
Data, code, and workflows should be available and cited Over the past two decades, computational methods have radically changed the ability of researchers from all areas of scholarship to process and analyze data and to simulate complex systems. But with these advances come challenges that are contributing to broader concerns over irreproducibility in the scholarly literature, among them the lack of transparency in disclosure of computational methods. Current reporting methods are often uneven… 
Reproducibility in Research: Systems, Infrastructure, Culture
TLDR
A high-level prototype open automated platform for scientific software development which effectively abstracts specific dependencies from the individual researcher and their workstation, allowing easy sharing and reproduction of results is proposed.
Publishing computational research - A review of infrastructures for reproducible and transparent scholarly communication
TLDR
A review of software solutions designed to solve the problem of publishing executable computational research results and properties related to preservation relevant for librarians dealing with long-term accessibility of research materials are included.
An empirical analysis of journal policy effectiveness for computational reproducibility
TLDR
This work evaluates the effectiveness of journal policy that requires the data and code necessary for reproducibility be made available postpublication by the authors upon request and finds it to be an improvement over no policy, but currently insufficient for reproducecibility.
Reproducible Research in Computational Chemistry of Materials
TLDR
The different aspects of research reproducibility in computational chemistry and materials science are summarized and the current practices of the community (both researchers and publishers) in this respect are delineated.
A Practical Taxonomy of Reproducibility for Machine Learning Research
TLDR
A taxonomy of reproducibility is discussed from the perspective of a practitioner who sets out to apply a new computational method within their own domain, those which provide the code, data, and full computational environment necessary to reproduce the results of the study.
A Practical Taxonomy of Reproducibility for Machine Learning Research
TLDR
A taxonomy of reproducibility is discussed from the perspective of a practitioner who sets out to apply a new computational method within their own domain, those which provide the code, data, and full computational environment necessary to reproduce the results of the study.
Epistemic issues in computational reproducibility: software as the elephant in the room
TLDR
It is argued that the epistemic issues at stake in actual practices of computational reproducedcibility are best unveiled by focusing on software as a pivotal concept, one that is surprisingly often overlooked in accounts of reproducibility issues.
Enabling the Verification of Computational Results: An Empirical Evaluation of Computational Reproducibility
  • V. Stodden, M. Krafczyk, A. Bhaskar
  • Computer Science
    Proceedings of the First International Workshop on Practical Reproducible Evaluation of Computer Systems
  • 2018
TLDR
This article attempts to regenerate published claims from author-supplied information, including data, code, inputs, and other provided specifications, on a different computational system than that used by the original authors.
Successes and Struggles with Computational Reproducibility: Lessons from the Fragile Families Challenge
TLDR
The authors describe their approach to enabling computational reproducibility for the 12 articles in this special issue of Socius about the Fragile Families Challenge, and draw on two tools commonly used by professional software engineers but not widely used by academic researchers: software containers and cloud computing.
...
...

References

SHOWING 1-10 OF 12 REFERENCES
Liberating field science samples and data
TLDR
Transparency and reproducibility enhance the integrity of research results for scientific and public uses and empower novel research applications and ways in which funders, publishers, scientific societies, and others are responding are discussed.
Best Practices for Computational Science: Software Infrastructure and Environments for Reproducible and Extensible Research
TLDR
A formalized set of best practice recommendations for computational scientists wishing to disseminate reproducible research, facilitate innovation by enabling data and code re-use, and enable broader communication of the output of digital scientific research are presented.
Quantifying Reproducibility in Computational Biology: The Case of the Tuberculosis Drugome
TLDR
This paper attempts to quantify this difficulty of reproducibility by reproducing a previously published paper for different classes of users (ranging from users with little expertise to domain experts) and suggest ways in which the situation might be improved.
The Legal Framework for Reproducible Scientific Research: Licensing and Copyright
  • V. Stodden
  • Computer Science
    Computing in Science & Engineering
  • 2009
TLDR
The author proposes the reproducible research standard for scientific researchers to use for all components of their scholarship that should encourage reproducible scientific investigation through attribution, facilitate greater collaboration, and promote engagement of the larger community in scientific learning and discovery.
Statistical Analyses and Reproducible Research
TLDR
This article describes a software framework for both authoring and distributing integrated, dynamic documents that contain text, code, data, and any auxiliary content needed to recreate the computations in data analyses, methodological descriptions, simulations, and so on.
Toward Reproducible Computational Research: An Empirical Analysis of Data and Code Policy Adoption by Journals
TLDR
A predictive model of open data and code policy adoption as a function of impact factor and publisher is built and it is found higher impact journals more likely to have Open Data and Code Policy Day and scientific societies more likely than commercial publishers to have openData and code policies.
Public Availability of Published Research Data in High-Impact Journals
TLDR
A substantial proportion of original research papers published in high-impact journals are either not subject to any data availability policies, or do not adhere to the data availability instructions in their respective journals.
Promoting an open research culture
Author guidelines for journals could help to promote transparency, openness, and reproducibility Transparency, openness, and reproducibility are readily recognized as vital features of science (1,
The Princeton companion to applied mathematics
TLDR
This volume introduces readers to applied mathematics and its uses; explains key concepts; describes important equations, laws, and functions; looks at exciting areas of research; covers modeling and simulation; explores areas of application; and more.
Best Practices for Computational Science: Software Infrastructure and Environments for Reproducible and Extensible Research
Scholarly dissemination and communication standards are changing to reflect the increasingly computational nature of scholarly research, primarily to include the sharing of the data and code
...
...