Stimulus Sampling and Social Psychological Experimentation

@article{Wells1999StimulusSA,
  title={Stimulus Sampling and Social Psychological Experimentation},
  author={Gary L Wells and Paul D. Windschitl},
  journal={Personality and Social Psychology Bulletin},
  year={1999},
  volume={25},
  pages={1115 - 1125}
}
The authors discuss the problem with failing to sample stimuli in social psychological experimentation. Although commonly construed as an issue for external validity, the authors emphasize how failure to sample stimuli also can threaten construct validity. They note some circumstances where the need for stimulus sampling is less obvious and more obvious, and they discuss some well-known cognitive biases that can contribute to the failure of researchers to see the need for stimulus sampling… 

Figures from this paper

Treating stimuli as a random factor in social psychology: a new and comprehensive solution to a pervasive but largely ignored problem.
TLDR
The substantial biases inherent in analyses that ignore one or the other of the random factors are shown, and the substantial advantages of the mixed models approach are illustrated with both hypothetical and actual, well-known data sets in social psychology.
The Limits of Direct Replications and the Virtues of Stimulus Sampling
While direct replications such as the ‘‘Many Labs’’ project are extremely valuable in testing the reliability of published findings across laboratories, they reflect the common reliance in psychology
The Limits of Direct Replications and the Virtues of Stimulus Sampling
TLDR
This work calls on editors to be aware of the challenges inherent in stimulus sampling, to expect and tolerate unexplained variability in observed effect size between stimuli, and to encourage stimulus sampling instead of the deceptively cleaner picture offered by the current reliance on single stimuli.
Replicating Studies in Which Samples of Participants Respond to Samples of Stimuli
TLDR
This paper discusses the use of both resampled and expanded stimulus sets, that is, stimulus samples that include the original stimuli plus new stimuli that are almost entirely based on a single stimulus sample that was later discovered to be highly unrepresentative.
Beware of samples! A cognitive-ecological sampling approach to judgment biases.
: A cognitive-ecological approach to judgment biases is presented and substantiated by recent empirical evidence. Latent properties of the environment are not amenable to direct assessment but have
A Closer Look at Social Psychologists’ Silver Bullet
  • H. Bless, Axel M. Burger
  • Psychology, Medicine
    Perspectives on psychological science : a journal of the Association for Psychological Science
  • 2016
TLDR
It is argued that it is exactly these elements that constitute core advantages of experimental research but that are—at the same time—associated with side effects, which are often out of focus when researchers derive theoretical conclusions from their experimental findings.
Are valence and social avoidance associated with the memory conformity effect?
TLDR
The memory conformity effect was larger for previously unseen stimuli (fillers) than for previously seen stimuli (targets), and was greatest for those with low scores on a social avoidance measure.
The Chicago face database: A free stimulus set of faces and norming data
TLDR
The Chicago Face Database is introduced, a free resource consisting of 158 high-resolution, standardized photographs of Black and White males and females between the ages of 18 and 40 years and extensive data about these targets and factors associated with researchers’ judgments of suitability.
Generalizing across stimuli as well as subjects: A non-mathematical tutorial on mixed-effects models
aDepartment of Psychology, Rice University bMultimodal Imaging Laboratory, University of California, San Diego cDepartment of Psychiatry, University of California Abstract Although it has long been
Meta-Cognitive Myopia and the Dilemmas of Inductive-Statistical Inference
TLDR
The central message is laid out that MM offers an alternative account of many biases in judgment and decision making, which have been traditionally explained in terms of capacity constraints, limited reasoning ability, motivational forces, or severely biased environmental input.
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 19 REFERENCES
The Category-Confound: A Design Error
Summary An experimental design problem, found to be frequent in some of the social psychology literature, occurs when only one sample from a population of possible samples is used to define a
The language-as-fixed-effect fallacy: A critique of language statistics in psychological research.
Current investigators of words, sentences, and other language materials almost never provide statistical evidence that their findings generalize beyond the specific sample of language materials they
BELIEF IN THE LAW OF SMALL NUMBERS
“Suppose you have run an experiment on 20 subjects, and have obtained a significant result which confirms your theory ( z = 2.23, p If you feel that the probability is somewhere around .85, you may
Construct validity in psychological tests.
TLDR
The present interpretation of construct validity is not "official" and deals with some areas where the Committee would probably not be unanimous, but the present writers are solely responsible for this attempt to explain the concept and elaborate its implications.
Quasi-Experimentation
"action research" for the less rigorous aspects of educational investigations. Will quasi-experimentation become another educational fad, misunderstood and misused for a while before being dropped in
The Perceived Credibility of Child Eyewitnesses: What Happens When They Use Their Own Words?
How do triers-of-fact judge the credibility of children versus that of adults as eyewitnesses? We argue that there are two processes to consider in answering this question and that previous research
Quantitative Methods in Social Psychology
T HE TITLE of this paper represents a small subdivision of one of the subtlest problems in the field of sociology, namely, the nature of the symbolic mechanisms through which man deals to an
Accuracy, confidence, and juror perceptions in eyewitness identification.
TLDR
Jurors were unable to distinguish accurate from inaccurate witnesses across the 42 cross-examina tion sessions, and jurors in the leading-questions conditions were significantly more likely to believe accurate than inaccurate witnesses, whereas the reverse effect held for nonleading questions.
The Effects of Graduate Training on Reasoning: Formal Discipline and Thinking About Everyday-Life Events
The theory of formal disciplinenthat is, the view that instruction in abstract rule systems can affect reasoning about everyday-life eventsnhas been rejected by 20th century psychologists on the
Essentials of Behavioral Research: Methods and Data Analysis
TLDR
This chapter discusses the nature of Behavioral Research, the development and testing of research ideas, and the selection of subjects and Stimuli for experiments.
...
1
2
...