Increasing the Transparency of Research Papers with Explorable Multiverse Analyses

@article{Dragicevic2019IncreasingTT,
  title={Increasing the Transparency of Research Papers with Explorable Multiverse Analyses},
  author={Pierre Dragicevic and Yvonne Jansen and Abhraneel Sarma and Matthew Kay and Fanny Chevalier},
  journal={Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems},
  year={2019}
}
We present explorable multiverse analysis reports, a new approach to statistical reporting where readers of research papers can explore alternative analysis options by interacting with the paper itself. This approach draws from two recent ideas: i) multiverse analysis, a philosophy of statistical reporting where paper authors report the outcomes of many different statistical analyses in order to show how fragile or robust their findings are; and ii) explorable explanations, narratives that can… 

Figures from this paper

A Survey of Tasks and Visualizations in Multiverse Analysis Reports
Analysing data from experiments is a complex, multi‐step process, often with multiple defensible choices available at each step. While analysts often report a single analysis without documenting how
Statistical Significance Testing at CHI PLAY: Challenges and Opportunities for More Transparency
TLDR
It is found that over half of these papers employ NHST without specific statistical hypotheses or research questions, which may risk the proliferation of false positive findings and provide a template for more transparent research and reporting practices.
Assessing the Robustness of Mediation Analysis Results Using Multiverse Analysis
TLDR
It is shown that multiverse analysis is a useful method for the assessment of the robustness of the direct, indirect, and total effect estimates in a mediation analysis and thereby to inform replication studies.
Paths Explored, Paths Omitted, Paths Obscured: Decision Points & Selective Reporting in End-to-End Data Analysis
TLDR
This study pore over nine published research studies and conducts semi-structured interviews with their authors to confirm that researchers may experiment with choices in search of desirable results, but also identify other reasons why researchers explore alternatives yet omit findings.
Seven steps toward more transparency in statistical practice.
TLDR
It is argued that statistical practice in the social and behavioural sciences benefits from transparency, a fair acknowledgement of uncertainty and openness to alternative interpretations, and that these ethical considerations-as well as their statistical consequences-establish common ground among data analysts, despite continuing disagreements about the foundations of statistical inference.
Consensus-based guidance for conducting and reporting multi-analyst studies
TLDR
How broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research is discussed.
Seven Steps Toward More Transparency in Statistical Practice
We argue that statistical practice in the social and behavioral sciences benefits from transparency, a fair acknowledgement of uncertainty, and openness to alternative interpretations. To promote
How Visualizing Inferential Uncertainty Can Mislead Readers About Treatment Effects in Scientific Results
TLDR
It is found that axis rescaling reduces error, but not as well as prediction intervals or animated hypothetical outcome plots (HOPs), and that depicting inferential uncertainty causes participants to underestimate variability in individual outcomes.
Data Comics for Reporting Controlled User Studies in Human-Computer Interaction
TLDR
This paper provides examples of data comics study reports alongside a graphical repertoire of examples, embedded in a framework of guidelines for creating comics reports which was iterated upon and evaluated through a series of collaborative design sessions.
Interaction Techniques for Visual Exploration Using Embedded Word-Scale Visualizations
TLDR
A design space of view manipulation interactions for small data-driven contextual visualizations (word-scale visualizations) that support an active reading experience and engage readers through exploration of embedded visualizations whose placement and content connect them to specific terms in a document is described.
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 129 REFERENCES
Increasing Transparency Through a Multiverse Analysis
TLDR
It is shown that analyzing a single data set can be misleading and a multiverse analysis is proposed as an alternative practice, which involves performing all analyses across the whole set of alternatively processed data sets corresponding to a large set of reasonable scenarios.
Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results
Twenty-nine teams involving 61 analysts used the same data set to address the same research question: whether soccer referees are more likely to give red cards to dark-skin-toned players than to
The New Statistics
TLDR
An eight-step new-statistics strategy for research with integrity is described, which starts with formulation of research questions in estimation terms, has no place for NHST, and is aimed at building a cumulative quantitative discipline.
Specification Curve: Descriptive and Inferential Statistics on All Reasonable Specifications
Empirical results often hinge on data analytic decisions that are simultaneously defensible, arbitrary, and motivated. To mitigate this problem we introduce Specification-Curve Analysis, which
Moving Transparent Statistics Forward at CHI
TLDR
A concentrated one-day writing workshop is proposed to develop rough drafts of reviewer guidelines, resources for authors, and other suggestions for advancing a vision of transparent statistics within the field.
Special Interest Group on Transparent Statistics Guidelines
TLDR
A SIG is proposed to solicit feedback from the CHI community on a first working draft of "Transparent Statistics Guidelines" and engage potential contributors to push the transparent statistics movement forward.
Researcher-Centered Design of Statistics: Why Bayesian Statistics Better Fit the Culture and Incentives of HCI
TLDR
It is shown that Bayesian analysis yields more precise effects with each new study, facilitating knowledge accrual without traditional meta-analyses, and Bayesian methods can be adopted from the bottom up without the need for new incentives for replication or meta-analysis.
A manifesto for reproducible science
TLDR
This work argues for the adoption of measures to optimize key elements of the scientific process: methods, reporting and dissemination, reproducibility, evaluation and incentives, in the hope that this will facilitate action toward improving the transparency, reproducible and efficiency of scientific research.
Hypothetical Outcome Plots Help Untrained Observers Judge Trends in Ambiguous Data
TLDR
This work identifies and motivate an appropriate task to investigate realistic judgments of uncertainty in the public domain through a qualitative analysis of uncertainty visualizations in the news, and contributes two crowdsourced experiments comparing the effectiveness of HOPs, error bars, and line ensembles for supporting perceptual decision-making from visualized uncertainty.
Statistical Analyses and Reproducible Research
TLDR
This article describes a software framework for both authoring and distributing integrated, dynamic documents that contain text, code, data, and any auxiliary content needed to recreate the computations in data analyses, methodological descriptions, simulations, and so on.
...
1
2
3
4
5
...