Evidence on “What Works”: An Argument for Extended-Term Mixed-Method (ETMM) Evaluation Designs

  title={Evidence on “What Works”: An Argument for Extended-Term Mixed-Method (ETMM) Evaluation Designs},
  author={Madhabi Chatterji},
  journal={Educational Researcher},
  pages={13 - 3}
Federal policy tools for gathering evidence on “What Works” in education, such as the What Works Clearinghouse’s (WWC) standards, emphasize randomized field trials as the preferred method for generating scientific evidence on the effectiveness of educational programs. This article argues instead for extended-term mixed-method (ETMM) designs. Emphasizing the need to consider temporal factors in gaining thorough understandings of programs as they take hold in organizational or community settings… Expand
Gathering evidence on an after-school supplemental instruction program: Design challenges and early findings in light of NCLB.
The No Child Left Behind (NCLB) Act of 2001 requires that public schools adopt research-supported programs and practices, with a strong recommendation for randomized controlled trials (RCTs) as theExpand
Grades of Evidence
This article argues with a literature review that a simplistic distinction between strong and weak evidence hinged on the use of randomized controlled trials (RCTs), the federal “gold standard” forExpand
An Analysis of the Fidelity Implementation Policies of the What Works Clearinghouse
Simulations show that the What Works Clearinghouse policy minimizes the positive impact of highly effective programs and the negative impact ofhighly ineffective programs. Expand
Causal inferences on the effectiveness of complex social programs: Navigating assumptions, sources of complexity and evaluation design challenges.
This paper explores avenues for navigating evaluation design challenges posed by complex social programs (CSPs) and their environments when conducting studies that call for generalizable, causalExpand
Lessons Learned from the Use of Randomized and Quasi-Experimental Field Designs for the Evaluation of Educational Programs.
Abstract As a result of the federal No Child Left Behind Act (NCLB) of 2002, the field of education has seen a heavy emphasis on the use of “scientifically based research” for designing and testingExpand
Conclusion: Agency in the face of complexity and the future of assumption-aware evaluation practice.
This final chapter in the volume pulls together common themes from the diverse set of articles by a group of eight authors in this issue, and presents some reflections on the next steps for improvingExpand
A model for fidelity of implementation in a study of a science curriculum unit: Evaluation based on program theory
This work was based on the work conducted by SCALE-uP: A collaboration between George Washington University and Montgomery County Public Schools (MD); Sharon Lynch, Joel Kuipers, Curtis Pyke, andExpand
The development of the concept of “evidence” in the U.S. educational policies
It has been approximately twenty years since the word “evidence” entered in educational research thanks to Evidence Based Education and a debate grew immediately cause to the complexity in definingExpand
“The Research is Continuing, Whether We Are There or Not”1. Research on Continuing Education Between Scientific Traditions and Evidence Expectations
Evidence-based governance – international and supranational stakeholders of educational policy such as the European Union (EU) or the Organisation for Economic Co-operation and Development (OECD)Expand
Building Confidence in Qualitative Research
The quality of qualitative research has been subject to considerable criticism recently, partly driven by the development of an international movement for “evidence-based policy and practice.” In theExpand


Models and Methods for Examining Standards-Based Reforms and Accountability Initiatives: Have the Tools of Inquiry Answered Pressing Questions on Improving Schools?
This article synthesizes research on standards-based reforms and accountability, with specific attention to purposes, models, and methods of inquiry. Starting with the premise that the reforms wereExpand
Comment: Culture, Rigor, and Science in Educational Research
In this article the authors argue that both the Feuer, Towne, and Shavelson article and the larger National Research Council (NRC) report on which it is based must be understood in the context ofExpand
Contestation and Change in National Policy on “Scientifically Based” Education Research
In this article, we examine the definitions of “scientifically based research” in education that have appeared in recent national legislation and policy. These definitions, now written into law inExpand
Randomized Experiments in Educational Policy Research: A Critical Examination of the Reasons the Educational Evaluation Community has Offered for not Doing Them
This article notes the paucity with which reform efforts in education have been evaluated experimentally, despite well-nigh universal acknowledgement that experiments provide the best justificationExpand
Scientific Culture and Educational Research
The No Child Left Behind Act of 2001 requires federal grantees to use their funds on evidence-based strategies. The law includes definitions of research quality, which are also featured prominentlyExpand
Advances in mixed-method evaluation : the challenges and benefits of integrating diverse paradigms
Evaluation theory and practice today are characteristically pluralistic, embracing diverse perspectives, methods, data, and values within and across studies that aim to generate more insightful andExpand
Beyond Use: Understanding Evaluation’s Influence on Attitudes and Actions
Although use is a core construct in the field of evaluation, neither the change processes through which evaluation affects attitudes, beliefs, and actions, nor the interim outcomes that lie betweenExpand
Transcending the Qualitative-Quantitative Debate: The Analytic and Systemic Approaches to Educational Research
There is a distinction to be made that transcends the one between the quantitative and qualitative research paradigms. It is a distinction between that kind of research that suits best the study ofExpand
Comment: “Science” Rejects Postmodernism
The National Research Council report Scientific Research in Educationclaims to present an inclusive view of science as it responds to federal government attempts to legislate educational research.Expand
Comment: Be Careful What You Wish For—You May Get It: Educational Research in the Spotlight
Two matters of deep concern in the Feuer, Towne, and Shavelson article are the focus of this commentary. First, the authors agree that legislative mandates reifying particular methodologicalExpand