Subgroup Analysis in Social Experiments: Measuring Program Impacts Based on Post-Treatment Choice

  title={Subgroup Analysis in Social Experiments: Measuring Program Impacts Based on Post-Treatment Choice},
  author={Laura R. Peck},
  journal={American Journal of Evaluation},
  pages={157 - 187}
  • Laura R. Peck
  • Published 1 June 2003
  • Computer Science
  • American Journal of Evaluation
A fundamental question within the field of program evaluation is “Do social programs work?” Although experiments allow us to answer this question with certainty, they have some limitations. Experiments generate mean program impacts and even mean impacts by subgroup, but they often leave unexplored the impacts on subgroups determined by treatment use. This work proposes a methodology for analyzing the impacts of social programs on previously unexamined subgroups. Rather than using a single trait… 

Figures and Tables from this paper

Using Cluster Analysis in Program Evaluation
This article explains the challenge of treatment group heterogeneity and proposes using cluster analysis to identify otherwise difficult-to-identify subgroups within evaluation data, thereby producing unbiased estimates of program impacts by subgroup.
Analyzing the Influence of Dosage in Social Experiments With Application to the Supporting Healthy Marriage Program
In experimental evaluations of health and social programs, the role of dosage is rarely explored because researchers cannot usually randomize individuals to experience varying dosage levels. Instead,
On the “How” of Social Experiments: Analytic Strategies for Getting Inside the Black Box
Analysis of postrandom assignment (endogenous) events or experiences in experimental evaluation data is becoming increasingly widespread. This chapter highlights some analytic strategies for
Privileging the Participant: The Importance of Sub-Group Analysis in Social Welfare Evaluations
This paper analyzes how variation in participant take-up rates affected the impacts of the New Hope project, a random-assignment, anti-poverty program. New Hope offered experimental members four
Conditions for Effective Application of Analysis of Symmetrically-Predicted Endogenous Subgroups
Several analytic strategies exist for opening up the “black box” to reveal more about what drives policy and program impacts. This article focuses on one of these strategies: the Analysis of
Using Symmetric Prediction of Endogenous Subgroups for Causal Inferences About Program Effects Under Robust Assumptions
To answer “what works?” questions about policy interventions based on an experimental design, Peck (2003) proposes to use baseline characteristics to symmetrically divide treatment and control group
On the Feasibility of Extending Social Experiments to Wider Applications
Background: When deciding how to allocate limited funds for social programs, policymakers and program managers increasingly ask for evidence of effectiveness based on studies that rely on solid
On Analysis of Symmetrically Predicted Endogenous Subgroups
Researchers and policy makers are increasingly dissatisfied with the “average treatment effect.” Not only are they interested in learning about the overall causal effects of policy interventions, but
What Are the Effects of Welfare Sanction Policies? Or, Using Propensity Scores as a Subgroup Indicator to Learn More From Social Experiments
This article uses propensity scores to identify subgroups of individuals most likely to experience a reduction in cash benefits because of sanctions in some of the programs that make up the National
Estimating Complier Average Causal Effects for Clustered RCTs When the Treatment Affects the Service Population
RCTs sometimes test interventions that aim to improve existing services targeted to a subset of individuals identified after randomization. Accordingly, the treatment could affect the composition of


Subgroup Impacts of Large-Scale Welfare Employment Programs
Recent research has suggested that employment programs for welfare recipients may be most effective if resources a re systematically targeted on those most likely to remain on public assistance a
Making the Most out of Programme Evaluations and Social Experiments: Accounting for Heterogeneity in Programme Impacts
The conventional approach to social programme evaluation focuses on estimating mean impacts of programmes. Yet many interesting questions regarding the political economy of programmes, the
The Distributional Impacts of Social Programs
We illustrate the use of nonparametnc quantile regression in examining the impacts of social programs on the distributions of noncategorical outcome measures. We apply the method to earnings effects
Modeling the Performance of Welfare-to-Work Programs: The Effects of Program Management and Services, Economic Environment, and Client Characteristics. MDRC Working Papers on Research Methodology.
This paper poses a question of direct relevance for welfare administrators, program operators, and policy makers: What management practices, program strategies, and local conditions are key to
Learning about Treatment Effects from Experiments with Random Assignment of Treatments
The importance of social programs to a diverse population creates a legitimate concern that the findings of evaluations be widely credible. The weaker the assumptions imposed, the more widely
The Impact of Being Offered and Receiving Classroom Training on the Employment Histories of Disadvantaged Women: Evidence from Experimental Data
We address two questions using experimental data on disadvantaged women. First, what is the impact of being offered JTPA classroom training on the duration of unemployment and employment? Second,
Variation in teenage mothers' experiences of child care and other components of welfare reform: selection processes and developmental consequences.
Examination of effects on 4- to 6-year-old children of different patterns of child care, self-sufficiency activities, and other service utilization indicators among experimental-group mothers in a 16-site welfare reform program found that children of mothers in groups characterized by high involvement in center-based care, education, and job training showed higher levels of cognitive ability than those with low levels of involvement.
Assessing the effect of an influenza vaccine in an encouragement design.
An extended framework for the analysis of data from randomized experiments with a binary treatment, binary encouragement, and background covariates is presented, suggesting that global intention-to-treat estimates are too coarse and even misleading when taken as summarizing the evidence in the data for the effects of treatments.
Accounting for No-Shows in Experimental Evaluation Designs
Examination of how to estimate the effect of a program in the presence of no- shows—persons who are assigned to the program but do not participate finds several estimators that overcome these problems.
Identification Problems in the Social Sciences.
Methodological research in the social sciences aims to learn what conclusions can and cannot be drawn given empirically relevant combinations of assumptions and data. Methodologists have long found