Promises and Perils of Experimentation: The Mutual-Internal-Validity Problem

@article{Lin2021PromisesAP,
  title={Promises and Perils of Experimentation: The Mutual-Internal-Validity Problem},
  author={Hause Lin and Kaitlyn M. Werner and Michael Inzlicht},
  journal={Perspectives on Psychological Science},
  year={2021},
  volume={16},
  pages={854 - 863}
}
Researchers run experiments to test theories, search for and document phenomena, develop theories, or advise policymakers. When testing theories, experiments must be internally valid but do not have to be externally valid. However, when experiments are used to search for and document phenomena, develop theories, or advise policymakers, external validity matters. Conflating these goals and failing to recognize their tensions with validity concerns can lead to problems with theorizing… 

Moral psychology from the lab to the wild: Relief registries as a paradigm for studying real-world altruism

Experimental psychology’s recent shift toward low-effort, high-volume methods (e.g., self-reports, online studies) and away from the more effortful study of naturalistic behavior raises concerns

Beyond Statistical Ritual: Theory in Psychological Science

  • T. ProulxR. Morey
  • Psychology
    Perspectives on psychological science : a journal of the Association for Psychological Science
  • 2021
TLDR
It is suggested that psychologists have too often chosen the latter option, substituting synthetic certainties for theory-guided research, in much the same manner as Scholastic scholars did centuries ago.

Whither Inhibition?

Inhibition is considered a process essential to goal pursuit and as a result has become a central construct in many disciplines in psychology and adjacent fields. Despite a century’s worth of debate,

Longitudinal evidence that Event Related Potential measures of self-regulation do not predict everyday goal pursuit

Self-regulation has been studied across levels of analysis; however, little attention has been paid to the extent to which self-report, neural, and behavioral indices predict goal pursuit in

Cognitive effort for self, strangers, and charities

Effort is aversive and often avoided, even when earning benefits for oneself. Yet, people sometimes work hard for others. How do people decide who is worth their effort? Prior work shows people avoid

Intolerance of Transgressive Protest Actions: The Differential Roles of Deontological and Utilitarian Morality.

The current research examines intolerance of protest actions by focusing on two major questions: (a) How intolerant are people of transgressive protest actions of their least-liked versus most-liked

Person- and variable-centred quantitative analyses in educational research: insights concerning Australian students’ and teachers’ engagement and wellbeing

Decreasing opportunities for education researchers in Australia to learn quantitative methods of analysis have been lamented by those who possess these skills and knowledges, and by policymakers

Cold Entropy: Assessing Individual Differences in Cognitive Adaptability during Cold Stress

Abstract Human cognition unfolds in a multitude of environments, including those that are associated with extreme stressors. Successfully measuring and modeling behaviors in such environments is

References

SHOWING 1-10 OF 149 REFERENCES

Artificiality: The tension between internal and external validity in economic experiments

The artificiality of a laboratory situation is placed in the context of the tension between external and internal validity. Most economists consider internal validity to be most important. A proper

Theory Before the Test: How to Build High-Verisimilitude Explanatory Theories in Psychological Science

TLDR
It is suggested that psychological science, by focusing on effects, may lose sight of its primary explananda: psychological capacities, and how to leverage effects to understand capacities better is explained.

What Do Laboratory Experiments Tell Us About the Real World

An important question facing experimental economists is whether behavior inside the laboratory is a good indicator of behavior outside the laboratory. We begin with a model that assumes the choices

Replicating patterns of prospect theory for decision under risk

TLDR
In a sample of over 4,000 participants from 19 countries, the core patterns from a highly influential study on behaviour and decision-making broadly replicate, with only minor exceptions and somewhat smaller effect sizes.

Experimental practices in economics: A methodological challenge for psychologists?

TLDR
There is a call for more research on the consequences of methodological preferences, such as the use on monetary payments, and a “do-it-both-ways” rule regarding the enactment of scripts, repetition of trials, and performance-based monetary payments.

The generalizability crisis

  • T. Yarkoni
  • Psychology
    Behavioral and Brain Sciences
  • 2020
TLDR
It is argued that failure to take the alignment between verbal and statistical expressions seriously lies at the heart of many of psychology's ongoing problems (e.g., the replication crisis), and concludes with a discussion of several potential avenues for improvement.

Experiments in economics: External validity and the robustness of phenomena

External validity is the problem of generalizing results from laboratory to non‐laboratory conditions. In this paper we review various ways in which the problem can be tackled, depending on the kind

In defense of external invalidity.

Many psychological investigations are accused of "failure to generalize to the real world" because of sample bias or artificiality of setting. It is argued in this article that such "generalizations"

Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology.

Abstract Theories in “soft” areas of psychology lack the cumulative character of scientific knowledge. They tend neither to be refuted nor corroborated, but instead merely fade away as people lose

Experimental Economics From the Vantage‐point of Behavioural Economics

As a behavioural economist (an economist who brings psychological insights to bear on economic phenomena), preparing a controversy corner piece criticising experimental economics (the use of
...