Improving Survey Methods With Cognitive Interviews in Small- and Medium-Scale Evaluations

  title={Improving Survey Methods With Cognitive Interviews in Small- and Medium-Scale Evaluations},
  author={Katherine E. Ryan and Nora Gannon-Slater and Michael J. Culbertson},
  journal={American Journal of Evaluation},
  pages={414 - 430}
Findings derived from self-reported, structured survey questionnaires are commonly used in evaluation and applied research to inform policy-making and program decisions. Although there are a variety of issues related to the quality of survey evidence (e.g., sampling precision), the validity of response processes–how respondents process thoughts and perceptions when answering questionnaires–is crucial. We assess the extent to which cognitive interviews, used to test survey questions as part of… 

Figures from this paper

Using the cognitive interviewing process to improve survey design by allied health: A qualitative study

The cognitive interviewing process applied during the development of a purpose-designed survey enabled the identification of potential problems and informed revisions to the survey prior to its use.

Cognitive Interviewing Methodologies

This editorial will build upon the International Society of Pharmacoeconomics and Outcomes Research (ISPOR PRO) Good Research Practices Task Force Report: Part 2, which provides useful general guidance on the application of CI in PROM development in order to provide the reader with some guidance in theApplication of CI for survey development.

School self-evaluation instruments and cognitive validity. Do items capture what they intend to?

ABSTRACT School self-evaluation (SSE) often makes use of questionnaires in order to sketch a picture of the school. How respondents cognitively process questionnaire items determines the validity of

Cognitive Interviews for Reviewing Multiple-Choice Items in Mathematics.

During the development of a test of multiple-choice items, cognitive interviews were conducted with students from lower secondary school. The purpose of these interviews was to confirm that the

Instruments for school self-evaluation: lost in translation? A study on respondents’ cognitive processing

School self-evaluation (SSE), as an important leverage for quality assurance, often relies on surveys among staff members to collect information on the schools’ functioning. The extent to which

Talking to People I: Surveys

Surveys are built up of standardized interviews with larger samples of individuals in order to make inferences about a specific population. They belong to the most popular methods within the social

Improving the Face Validity of Self-Report Scales through Cognitive Interviews Based on Tourangeau Question and Answer Framework: A Practical Work on the Nursing Talent Identification Scale

Cognitive interview was effective in identifying problematic items of the Nursing Talent Identification Scale and how respondents understand and answer to items through cognitive interviews is determined to assess the user-friendliness of the scale and increase its face validity.

Employing cognitive interviewing to evaluate, improve and validate items for measuring the health-related quality of life of women diagnosed with ovarian cancer

The cognitive interviewing process established content validity for the tool, thereby, preparing it for field testing and evaluation of its psychometric properties and creating a health-related quality of life tool specific to women diagnosed with ovarian cancer.

Use of Cognitive Interviews to Test the Validity of a Reading Motivation Instrument for English Learners

Abstract This cognitive interview study aims to improve the content validity of a newly developed reading motivation instrument for English learners. Previous research has shown there is a

Development of the Survey of Parent/Provider Expectations and Knowledge (SPEAK)

This study reported the development and initial validation of the Survey of Parent/Provider Expectations and Knowledge (SPEAK), a self-administered questionnaire assessing expectations and knowledge



Are We Asking the Right Questions? Using Cognitive Interviews to Improve Surveys in Education Research

Improving the validity and reliability of surveys is a critical part of the response to the call for improved rigor of education research, policy analysis and evaluation. Too often we create inquiry

Research Synthesis: The Practice of Cognitive Interviewing

Cognitive interviewing has emerged as one of the more prominent methods for identifying and correcting problems with survey questions. We define cognitive interviewing as the administration of draft

Methods for Testing and Evaluating Survey Questionnaires

Contributors. Preface. 1. Methods for Testing and Evaluating Survey Questions (Stanley Presser, et al.). PART I: COGNITIVE INTERVIEWS. 2. Cognitive Interviewing Revisited: A Useful Technique, in

Sources of Error in Cognitive Interviews

Cognitive interviewing is used to identify problems in questionnaires under development by asking a small number of pretest participants to verbally report their thinking while answering the draft

Efficient Respondent Selection for Cognitive Interviewing

This paper explores a seldom considered area of cognitive interview pretesting best practices: sample selection. The questions we consider are: Do different types of respondents produce different

Cognitive interviewing methodology in the development of a pediatric item bank: a patient reported outcomes measurement information system (PROMIS) study

The cognitive interviewing procedures employed by the Patient Reported Outcomes Measurement Information System (PROMIS) pediatrics group for the purpose of developing a dynamic, electronic item bank for field testing with children and adolescents using novel computer technology are discussed.

The accessibility and applicability of knowledge: predicting context effects in national surveys.

Using data from the National Health Interview Survey on Disability from 1994 and 1995, this research demonstrates that the size of accessibility effects can be modeled as a function of the applicability of the initial to the subsequent questions.

Designing Surveys: A Guide to Decisions and Procedures

This book discusses how to select a sample, design the Questionnaire, and reduce sources of Error in Data Collection using Census Data.

Measuring Instructional Practice: Can Policymakers Trust Survey Data?

Policymakers' new interest in reforming teaching has created a demand for accurate data on instructional practice. Most such data come from teacher surveys, although the accuracy of these surveys in

Questionnaire Design from a Cross-Cultural Perspective: An Empirical Investigation of Koreans and Non-Koreans

This research study investigated the cross-cultural functioning of health-survey questions presented to four groups: Koreans who were monolingual in Korean; non-Korean native speakers of English; bilingual Koreans interviews in English, and bilingual Koreans interviewed in Korean.