HARK No More: On the Preregistration of CHI Experiments

@article{Cockburn2018HARKNM,
  title={HARK No More: On the Preregistration of CHI Experiments},
  author={Andy Cockburn and Carl Gutwin and Alan J. Dix},
  journal={Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems},
  year={2018}
}
  • A. Cockburn, C. Gutwin, A. Dix
  • Published 19 April 2018
  • Psychology
  • Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems
Experimental preregistration is required for publication in many scientific disciplines and venues. When experimental intentions are preregistered, reviewers and readers can be confident that experimental evidence in support of reported hypotheses is not the result of HARKing, which stands for Hypothesising After the Results are Known. We review the motivation and outcomes of experimental preregistration across a variety of disciplines, as well as previous work commenting on the role of… 
Apéritif: Scaffolding Preregistrations to Automatically Generate Analysis Code and Methods Descriptions
TLDR
Apéritif, a research prototype that scaffolds the preregistration process and automatically generates analysis code and a methods description, is developed that reduces the effort of preregistering a study, facilitates researchers’ workflows, and promotes consistency between research artifacts.
Statistics for HCI
  • A. Dix
  • Psychology
    CHI Extended Abstracts
  • 2021
TLDR
This course aims to give attendees an understanding of the meaning of the various statistics they see in papers or need to use in their own work, and it will also offer the attendees an introduction to some of the instructor’s extensive additional online material.
Supporting the Design and Analysis of HCI Experiments
TLDR
This dissertation contributes an empirical understanding of how researchers design and analyze experiments, as well as a set of tools that support researchers during that process.
A Meta-Analysis of Effect Sizes of CHI Typing Experiments
TLDR
This work presents the reference for small, medium, and large effect sizes for typing experiments based on a meta-analysis of well-cited papers from CHI conference, which can be used to conduct a priori power analysis or assess the magnitude of the found effect.
Preregistration Is Neither Sufficient nor Necessary for Good Science
To address widespread perceptions of a reproducibility crisis in the social sciences, a growing number of scholars recommend the systematic preregistration of empirical studies. The purpose of this
"Is IEEE VIS *that* good?" On key factors in the initial assessment of manuscript and venue quality
TLDR
It is found that publication venue plays the biggest part in how visualization researchers assess research articles, and rating systems and metrics are least important criteria for researchers when assessing the quality of a venue.
Interactive Tools for Reproducible Science - Understanding, Supporting, and Motivating Reproducible Science Practices
TLDR
This thesis paves new ways for interaction with RDM tools that support and motivate reproducible science, and advocates the unique role of HCI in supporting, motivating, and transforming reproducible research practices through the design of tools that enable effective RDM.
Touchstone2: An Interactive Environment for Exploring Trade-offs in HCI Experiment Design
TLDR
An interactive environment for manipulating experiment design parameters, revealing patterns in trial tables, and estimating and comparing statistical power is developed and TSL, a declarative language that precisely represents experiment designs, is developed.
Interactive tools for reproducible science
TLDR
An empirical understanding of reproducible research practices and the role of supportive tools through research in HEP and across a variety of scientific fields is built and the term secondary usage forms of RDM tools is introduced.
Anchoring Effects and Troublesome Asymmetric Transfer in Subjective Ratings
TLDR
Four analyses of NASA-TLX data from four previously published HCI papers are reported on, finding that asymmetric transfer is common, occurring in 42% of tests analysed and the data conforms to predictions of anchoring effects.
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 48 REFERENCES
HCI... not as it should be: inferential statistics in HCI research
TLDR
The use of inferential statistics over the last two BCS HCI conferences and the last year of two leading HCI journals is surveyed and approaches for improving the use of statistics in HCI are considered.
HCI... not as it should be: inferential statistics in HCI research
TLDR
The use of inferential statistics over the last two BCS HCI conferences and the last year of two leading HCI journals is surveyed and approaches for improving the use of statistics in HCI are considered.
RepliCHI: the workshop
TLDR
The aim of this workshop is to trial a new venue that embodies the plans made in previous SIGs and panels, such that people can begin to give people an outlet to publish experiences of attempting to replicate HCI research, and challenge or confirm its findings.
Fair Statistical Communication in HCI
TLDR
This chapter explains in a non-technical manner why it would be beneficial for HCI to switch to an estimation approach, i.e., reporting informative charts with effect sizes and interval estimates, and offering nuanced interpretations of the results.
Is once enough?: on the extent and content of replications in human-computer interaction
TLDR
What the results mean to HCI, including how reporting of studies could be improved and how conferences/journals may change author instructions to get more replications, is discussed.
Researcher-Centered Design of Statistics: Why Bayesian Statistics Better Fit the Culture and Incentives of HCI
TLDR
It is shown that Bayesian analysis yields more precise effects with each new study, facilitating knowledge accrual without traditional meta-analyses, and Bayesian methods can be adopted from the bottom up without the need for new incentives for replication or meta-analysis.
The file drawer problem and tolerance for null results
TLDR
Quantitative procedures for computing the tolerance for filed and future null results are reported and illustrated, and the implications are discussed.
To Replicate or Not to Replicate?
TLDR
This work aims to explore how people use and interact with their mobile devices and understand how those interactions evolve over time, and to replicate studies of mobile device usage in the field.
Publication bias in the social sciences: Unlocking the file drawer
TLDR
Fully half of peer-reviewed and implemented social science experiments are not published, providing direct evidence of publication bias and identifying the stage of research production at which publication bias occurs: Authors do not write up and submit null findings.
Special Interest Group on Transparent Statistics in HCI
TLDR
A SIG is proposed to discuss problems and limitations in statistical practices in HCI and options for moving the field towards clearer and more reliable ways of writing about experiments.
...
1
2
3
4
5
...