Evidence on “What Works”: An Argument for Extended-Term Mixed-Method (ETMM) Evaluation Designs

  title={Evidence on “What Works”: An Argument for Extended-Term Mixed-Method (ETMM) Evaluation Designs},
  author={Madhabi Chatterji},
  journal={Educational Researcher},
  pages={13 - 3}
Federal policy tools for gathering evidence on “What Works” in education, such as the What Works Clearinghouse’s (WWC) standards, emphasize randomized field trials as the preferred method for generating scientific evidence on the effectiveness of educational programs. This article argues instead for extended-term mixed-method (ETMM) designs. Emphasizing the need to consider temporal factors in gaining thorough understandings of programs as they take hold in organizational or community settings… 

Gathering evidence on an after-school supplemental instruction program: Design challenges and early findings in light of NCLB.

The No Child Left Behind (NCLB) Act of 2001 requires that public schools adopt research-supported programs and practices, with a strong recommendation for randomized controlled trials (RCTs) as the

Grades of Evidence

This article argues with a literature review that a simplistic distinction between strong and weak evidence hinged on the use of randomized controlled trials (RCTs), the federal “gold standard” for

An Analysis of the Fidelity Implementation Policies of the What Works Clearinghouse

Simulations show that the What Works Clearinghouse policy minimizes the positive impact of highly effective programs and the negative impact ofhighly ineffective programs.

A model for fidelity of implementation in a study of a science curriculum unit: Evaluation based on program theory

This work was based on the work conducted by SCALE-uP: A collaboration between George Washington University and Montgomery County Public Schools (MD); Sharon Lynch, Joel Kuipers, Curtis Pyke, and

The development of the concept of “evidence” in the U.S. educational policies

It has been approximately twenty years since the word “evidence” entered in educational research thanks to Evidence Based Education and a debate grew immediately cause to the complexity in defining

“The Research is Continuing, Whether We Are There or Not”1. Research on Continuing Education Between Scientific Traditions and Evidence Expectations

Evidence-based governance – international and supranational stakeholders of educational policy such as the European Union (EU) or the Organisation for Economic Co-operation and Development (OECD)

Building Confidence in Qualitative Research

The quality of qualitative research has been subject to considerable criticism recently, partly driven by the development of an international movement for “evidence-based policy and practice.” In the

Scientific-Based Research and Randomized Controlled Trials, the “Gold” Standard? Alternative Paradigms and Mixed Methodologies

This article addresses three controversial issues related to mixed methods research and policy. First, “Scientific-Based Research” promoted by “No Child Left Behind” (NCLB) reinforces diametrically



Models and Methods for Examining Standards-Based Reforms and Accountability Initiatives: Have the Tools of Inquiry Answered Pressing Questions on Improving Schools?

This article synthesizes research on standards-based reforms and accountability, with specific attention to purposes, models, and methods of inquiry. Starting with the premise that the reforms were

Comment: Culture, Rigor, and Science in Educational Research

In this article the authors argue that both the Feuer, Towne, and Shavelson article and the larger National Research Council (NRC) report on which it is based must be understood in the context of

Contestation and Change in National Policy on “Scientifically Based” Education Research

In this article, we examine the definitions of “scientifically based research” in education that have appeared in recent national legislation and policy. These definitions, now written into law in

Scientific Culture and Educational Research

The No Child Left Behind Act of 2001 requires federal grantees to use their funds on evidence-based strategies. The law includes definitions of research quality, which are also featured prominently

Advances in mixed-method evaluation : the challenges and benefits of integrating diverse paradigms

Evaluation theory and practice today are characteristically pluralistic, embracing diverse perspectives, methods, data, and values within and across studies that aim to generate more insightful and

Transcending the Qualitative-Quantitative Debate: The Analytic and Systemic Approaches to Educational Research

There is a distinction to be made that transcends the one between the quantitative and qualitative research paradigms. It is a distinction between that kind of research that suits best the study of

Comment: Be Careful What You Wish For—You May Get It: Educational Research in the Spotlight

Two matters of deep concern in the Feuer, Towne, and Shavelson article are the focus of this commentary. First, the authors agree that legislative mandates reifying particular methodological

Stake's responsive evaluation: Core ideas and evolution

Responsive evaluation, as a doctrine extending and disciplining common sense, has an intellectual history, some of it passing through Robert Stake's work in the late sixties. That is where this

[Case study methods].

The purpose of conducting state case studies was to describe the current status of juvenile justice education, and ways in which NCLB has impacted these systems and services. The case studies had

Addressing sponsor and stakeholder needs in the evaluation authorizing environment: Trends and implications

The editors make note of several trends and implications arising from the discussions in the earlier chapters, in particular, the need to recognize the complexity of evaluation contexts involving