How Much Should We Trust


Most papers that employ Differences-in-Differences estimation (DD) use many years of data and focus on serially correlated outcomes but ignore that the resulting standard errors are inconsistent. To illustrate the severity of this issue, we randomly generate placebo laws in state-level data on female wages from the Current Population Survey. For each law, we use OLS to compute the DD estimate of its “effect” as well as the standard error of this estimate. These conventional DD standard errors severely understate the standard deviation of the estimators: we find an “effect” significant at the 5 percent level for up to 45 percent of the placebo interventions. We use Monte Carlo simulations to investigate how well existing methods help solve this problem. Econometric corrections that place a specific parametric form on the time-series process do not perform well. Bootstrap (taking into account the auto-correlation of the data) works well when the number of states is large enough. Two corrections based on asymptotic approximation of the variance-covariance matrix work well for moderate numbers of states and one correction that collapses the time series information into a “pre” and “post” period and explicitly takes into account the effective sample size works well even for small numbers of states. ∗We thank Lawrence Katz (the editor), three anonymous referees, Alberto Abadie, Daron Acemoglu, Joshua Angrist, Abhijit Banerjee, Victor Chernozhukov, Michael Grossman, Jerry Hausman, Kei Hirano, Bo Honore, Guido Imbens, Jeffrey Kling, Kevin Lang, Steven Levitt, Kevin Murphy, Ariel Pakes, Emmanuel Saez, Douglas Staiger, Robert Topel, Whitney Newey and seminar participants at Harvard, Massachusetts Institute of Technology, University of Chicago Graduate School of Business, University of California at Los Angeles, University of California Santa Barbara, Princeton University and University of Texas at Austin for many helpful comments. Tobias Adrian, Shawn Cole, and Francesco Franzoni provided excellent research assistance. We are especially grateful to Khaled for motivating us to write this paper. e-mail:;;

6 Figures and Tables

Citations per Year

198 Citations

Semantic Scholar estimates that this publication has 198 citations based on the available data.

See our FAQ for additional information.

Cite this paper

@inproceedings{Bertrand2003HowMS, title={How Much Should We Trust}, author={Marianne Bertrand and Esther Duflo and Sendhil Mullainathan and Michael D. Grossman and Jerry Hausman and Kei Hirano and Bo E. Honor{\'e} and Jeffrey R. Kling and K. Brandon Lang and Steven D . Levitt and Kevin Murphy and Ariel Pakes and Emmanuel Saez and Douglas O . Staiger}, year={2003} }