The Design of Field Experiments With Survey Outcomes: A Framework for Selecting More Efficient, Robust, and Ethical Designs

@article{Broockman2017TheDO,
  title={The Design of Field Experiments With Survey Outcomes: A Framework for Selecting More Efficient, Robust, and Ethical Designs},
  author={David E. Broockman and Joshua L. Kalla and Jasjeet S. Sekhon},
  journal={Political Analysis},
  year={2017},
  volume={25},
  pages={435 - 464}
}
There is increasing interest in experiments where outcomes are measured by surveys and treatments are delivered by a separate mechanism in the real world, such as by mailers, door-to-door canvasses, phone calls, or online ads. However, common designs for such experiments are often prohibitively expensive, vulnerable to bias, and raise ethical concerns. We show how four methodological practices currently uncommon in such experiments have previously undocumented complementarities that can… 
Increasing Response Rates and Representativeness of Online Panels Recruited by Mail: Evidence from Experiments in 12 Original Surveys
TLDR
This paper presents new evidence on best practices for an emerging panel survey approach: recruiting online panels by mail from defined sampling frames, and presents results from 12 original panel studies involving over 1 million individuals.
How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It
In principle, experiments o↵er a straightforward method for social scientists to accurately estimate causal e↵ects. However, scholars often unwittingly distort treatment e↵ect estimates by
Replicable quality across every channel: the Busara research agenda on methods in the Global South
TLDR
This work will examine what methodological practices work best for various populations, especially those with the least social power, in the Global South, to maintain data quality, and find Busara is well-positioned to do this.
Can Video Advertisements Change Partisanship? Results from Panel Survey Experiments
UPDATE: Please see updated paper for a description of a differential attrition problem that will require changes to the analysis and further experimentation.<br><br><br>Although partisan orientations
A Systematic Review of Field Experiments in Public Administration
TLDR
How field experiments developed over time is discussed and trends in field experimentation in public administration are highlighted and a future research agenda for public administration field experiments is suggested.
Do Online Voter Guides Empower Citizens? Evidence from a Field Experiment with Digital Trace Data
Voting Advice Applications (VAAs), which provide citizens with information on the party that best represents their political preferences, are often cited as evidence of the empowering capabilities of
Seeing the World Through the Other's Eye: An Online Intervention Reducing Ethnic Prejudice
We report the results of an intervention that targeted anti-Roma sentiment in Hungary using an online perspective-taking game. We evaluated the impact of this intervention using a randomized
Active matrix factorization for surveys
TLDR
This work proposes to ask fewer questions of respondents and impute missing responses via probabilistic matrix factorization, and obtains principled suggestions of questions to retain and can automate the design of shorter instruments.
“Outside Lobbying” over the Airwaves: A Randomized Field Experiment on Televised Issue Ads
We present the first field experiment on how organized interest groups’ television ads affect issue opinions. We randomized 31,404 voters to three weeks of interest group ads about either immigration
Reducing Exclusionary Attitudes through Interpersonal Conversation: Evidence from Three Field Experiments
Exclusionary attitudes—prejudice toward outgroups and opposition to policies that promote their well-being—are presenting challenges to democratic societies worldwide. Drawing on insights from
...
...

References

SHOWING 1-10 OF 110 REFERENCES
Causal Inference in Conjoint Analysis: Understanding Multidimensional Choices via Stated Preference Experiments
Survey experiments are a core tool for causal inference. Yet, the design of classical survey experiments prevents them from identifying which components of a multidimensional treatment are
Innovations in Experimental Design in Attitude Surveys
In the last decade a revolution has occurred in the design of public opinion surveys. The principal breakthrough has been to combine the distinctive external validity advantages of the representative
Effects of Telephone Canvassing on Turnout and Preferences: A Field Experiment
IN DECIDING how to allocate campaign resources, strategists have remarkably little scholarly research available on the effects of particular kinds of campaign activities. The large volume of
Durably reducing transphobia: A field experiment on door-to-door canvassing
TLDR
It is shown that a single approximately 10-minute conversation encouraging actively taking the perspective of others can markedly reduce prejudice for at least 3 months, and this potential is illustrated with a door-to-door canvassing intervention in South Florida targeting antitransgender prejudice.
How Face-to-Face Interviews and Cognitive Skill Affect Item Non-Response: A Randomized Experiment Assigning Mode of Interview
Technology and the decreased cost of survey research have made it possible for researchers to collect data using new and varied modes of interview. These data are often analyzed as if they were
Scalable Protocols Offer Efficient Design for Field Experiments
TLDR
This article proves that scalable protocols conserve statistical efficiency in the face of problems implementing the treatment regime, and three examples taken from voter mobilization field experiments demonstrate the utility of the design principles discussed.
From SATE to PATT : Combining Experimental with Observational Studies to Estimate Population Treatment Effects ∗
Randomised controlled trials (RCTs) can provide unbiased estimates of sample average treatment effects. However, a common concern is that RCTs often fail to provide unbiased estimates of population
How Large and Long-lasting Are the Persuasive Effects of Televised Campaign Ads? Results from a Randomized Field Experiment
We report the results of the first large-scale experiment involving paid political advertising. During the opening months of a 2006 gubernatorial campaign, approximately $2 million of television and
Moderation from Bias: A Field Experiment on Partisan Media in a New Democracy
Partisan media are often blamed for polarization in newly liberalized regimes. However, there is little empirical work on the subject, and information-processing theories suggest that extreme
...
...