A Cautionary Note on the Use of Matching to Estimate Causal Effects: An Empirical Example Comparing Matching Estimates to an Experimental Benchmark

@article{Arceneaux2010ACN,
  title={A Cautionary Note on the Use of Matching to Estimate Causal Effects: An Empirical Example Comparing Matching Estimates to an Experimental Benchmark},
  author={Kevin Arceneaux and Alan S. Gerber and Donald P. Green},
  journal={Sociological Methods \& Research},
  year={2010},
  volume={39},
  pages={256 - 282}
}
In recent years, social scientists have increasingly turned to matching as a method for drawing causal inferences from observational data. Matching compares those who receive a treatment to those with similar background attributes who do not receive a treatment. Researchers who use matching frequently tout its ability to reduce bias, particularly when applied to data sets that contain extensive background information. Drawing on a randomized voter mobilization experiment, the authors compare… Expand

Tables from this paper

Optimizing matching and analysis combinations for estimating causal effects
TLDR
Simulation results indicate that combining full matching with double robust analysis performed best in both the simulations and the applied example, particularly when combined with machine learning estimation methods. Expand
Dude, Where’s My Treatment Effect? Errors in Administrative Data Linking and the Destruction of Statistical Power in Randomized Experiments
TLDR
It is demonstrated that probabilistic linking substantially outperforms stringent linking criteria and allows researchers to recover a considerable share of the statistical power lost under stringent data-linking rules. Expand
Performance of Matching Methods to Unmatched Ordinary Least Squares Regression Under Constant Effects.
TLDR
Comparing inferences from propensity score matching, coarsened exact matching, and un-matched covariate-adjusted ordinary least squares regression (OLS) to identify which methods produced unbiased inferences at the expected type I error rate suggests when estimates from matching and OLS are similar, OLS inferences are unbiased more often than matching inferences. Expand
Causal Effects Under Self-Selection
The randomized assignment of units to treatment status provided by an experiment provides unparalleled leverage to understand the average causal effect of an assigned treatment. Yet politicalExpand
A Comparison of Approaches to Advertising Measurement: Evidence from Big Field Experiments at Facebook
TLDR
It is suggested that commonly used observational approaches based on the data usually available in the advertising industry often fail to accurately measure the true effect of advertising, and the incremental explanatory power the authors' data would require to enable observational methods to successfully measure advertising effects. Expand
Advancements for Functional Form Robustness
Social scientists face a dual problem of model uncertainty and methodological abundance. There are many different ways to conduct an analysis, but the true model is unknown. This uncertainty amongExpand
Administrative Data Linking and Statistical Power Problems in Randomized Experiments
TLDR
This paper derives an analytic result for the consequences of linking errors on statistical power and shows how the problem varies across different combinations of relevant inputs, including the matching error rate, the outcome density and the sample size. Expand
Empirical Performance of Covariates in Education Observational Studies
ABSTRACT This article summarizes results from 12 empirical evaluations of observational methods in education contexts. We look at the performance of three common covariate-types in observationalExpand
Bias Reduction in Quasi-Experiments With Little Selection Theory but Many Covariates
Abstract: In observational studies, selection bias will be completely removed only if the selection mechanism is ignorable, namely, all confounders of treatment selection and potential outcomes areExpand
Assessing Correspondence Between Experimental and Nonexperimental Estimates in Within-Study Comparisons
TLDR
Different distance-based correspondence measures for assessing correspondence in experimental and nonexperimental estimates are examined and a new and straightforward approach that combines traditional significance testing and equivalence testing in the same framework is recommended. Expand
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 72 REFERENCES
Comparing Experimental and Matching Methods Using a Large-Scale Voter Mobilization Experiment
TLDR
Testing the performance of matching by gauging the success with which matching approximates experimental results shows that brief paid get-out-the-vote phone calls do not increase turnout, while matching and regression show a large and significant effect. Expand
Characterizing Selection Bias Using Experimental Data
TLDR
Semiparametric econometric methods are applied to estimate the form of selection bias that arises from using nonexperimental comparison groups to evaluate social programs and to test the identifying assumptions that justify three widely-used classes of estimators. Expand
Matching as Nonparametric Preprocessing for Reducing Model Dependence in Parametric Causal Inference
TLDR
A unified approach is proposed that makes it possible for researchers to preprocess data with matching and then to apply the best parametric techniques they would have used anyway and this procedure makes parametric models produce more accurate and considerably less model-dependent causal inferences. Expand
The bias due to incomplete matching.
TLDR
A practical example shows that the bias due to incomplete matching can be severe, and moreover, can be avoided entirely by using an appropriate multivariate nearest available matching algorithm, which, in the example, leaves only a small residual biasDue to inexact matching. Expand
Practical propensity score matching: a reply to Smith and Todd
Abstract This paper discusses propensity score matching in the context of Smith and Todd's (Does matching overcome Lalonde's critique of nonexperimental estimators, J. Econom., in press) reanalysisExpand
Matching As An Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Programme
This paper considers whether it is possible to devise a nonexperimental procedure for evaluating a prototypical job training programme. Using rich nonexperimental data, we examine the performance ofExpand
Estimating causal effects of treatments in randomized and nonrandomized studies.
A discussion of matching, randomization, random sampling, and other methods of controlling extraneous variation is presented. The objective is to specify the benefits of randomization in estimatingExpand
Matching Estimators of Causal Effects
As the counterfactual model of causality has increased in popularity, sociologists have returned to matching as a research methodology. In this article, advances over the past two decades in matchingExpand
Identification of Causal Effects Using Instrumental Variables
TLDR
It is shown that the instrumental variables (IV) estimand can be embedded within the Rubin Causal Model (RCM) and that under some simple and easily interpretable assumptions, the IV estimand is the average causal effect for a subgroup of units, the compliers. Expand
Detecting selection bias, using propensity score matching, and estimating treatment effects: an application to the private returns to a master’s degree
Most research in the area of higher education is plagued by the problem of endogeneity or self-selection bias. Unlike ordinary least squares (OLS) regression, propensity score matching addresses theExpand
...
1
2
3
4
5
...