Reliability in Coding Open-Ended Data: Lessons Learned from HIV Behavioral Research

  title={Reliability in Coding Open-Ended Data: Lessons Learned from HIV Behavioral Research},
  author={Daniel J. Hruschka and Deborah J. Schwartz and Daphne Cobb St.John and Erin Picone-Decaro and Richard A. Jenkins and James W. Carey},
  journal={Field Methods},
  pages={307 - 331}
Analysis of text from open-ended interviews has become an important research tool in numerous fields, including business, education, and health research. Coding is an essential part of such analysis, but questions of quality control in the coding process have generally received little attention. This article examines the text coding process applied to three HIV-related studies conducted with the Centers for Disease Control and Prevention considering populations in the United States and Zimbabwe… 

Figures and Tables from this paper

Process guidelines for establishing Intercoder Reliability in qualitative studies

Qualitative interviews are increasingly being utilized within the context of intervention trials. While there is emerging assistance for conducting and reporting qualitative analysis, there are

Adapting HIV prevention evidence-based interventions in practice settings: an interview study

Examining multiple intervention adaptations made during pre-implementation, implementation, maintenance, and evolution phases of human immunodeficiency virus HIV prevention technology transfer found providing strategic technical assistance and written guidance can facilitate effective HIV preventionTechnology transfer of evidence-based interventions.

Coding In-depth Semistructured Interviews

This article presents a procedure for developing coding schemes for in-depth semistructured interview transcripts that involves standardizing the units of text on which coders work and improving the coding scheme’s discriminant capability to an acceptable point.

Guidelines for Establishing Reliability When Coding Narrative Data

Best practices for coding and establishing reliability when working with narrative data are reviewed, highlighting how establishing reliability must be seen as an evolving process, rather than simply a focus on the end product.

A Team-based Approach to Open Coding: Considerations for Creating Intercoder Consensus

This article reports on one approach for creating consensus when open coding within an iterative-inductive analytical strategy used in a variety of settings to foster efficient and credible analysis of larger qualitative data sets, particularly useful in applied research settings where rapid results are often required.

Preparing for analysis: a practical guide for a critical step for procedural rigor in large-scale multisite qualitative research studies

Despite attention to technical aspects of conducting qualitative research, few studies describe procedures for managing pre-analysis activities in large-scale multisite qualitative projects. As part

Preparing for analysis: a practical guide for a critical step for procedural rigor in large-scale multisite qualitative research studies

Two exemplary approaches to pre-analysis are described, with emphasis on lessons learned and recommendations for coordinating resources, managing data, and maintaining fidelity to the study procedures for large-scale qualitative projects.

Observational Coding Training Methods for CBT Treatment Fidelity: A Systematic Review

Observational coding is the gold standard for coding treatment fidelity in Cognitive Behavioral Therapy (CBT). This method allows researchers to objectively assess therapist behaviors to determine if

Examining Fidelity of Web-based Acceptance and Commitment Interventions for Women with Chronic Widespread Pain

This study indicates that ACT can reliably be delivered in a written web-based format and based on the level of adherence to ACT-principles the treatment integrity can be judged as high.

HIV provider and patient perspectives on the Development of a Health Department “Data to Care” Program: a qualitative study

Health departments can build support for Data to Care efforts by gathering input of key stakeholders, such as HIV medical and social service providers, and coordinating with clinic-based efforts to re-engage patients in care.



Intercoder Agreement in Analysis of Responses to Open-Ended Interview Questions: Examples from Tuberculosis Research

Social scientists commonly conduct surveys to learn about beliefs, attitudes, reported behaviors, or experiences prevalent in a population. These interviews often involve the use of structured

The Place of Inter-Rater Reliability in Qualitative Research: An Empirical Study

Assessing inter-rater reliability, whereby data are independently coded and the codings compared for agreements, is a recognised process in quantitative research. However, its applicability to

Fixed-Choice and Open-Ended Response Formats: A Comparison from HIV Prevention Research in Zimbabwe

This article examined similarities and differences in responses to open-ended and fixed-choice question formats gathered from Zimbabwean women participating in an HIV intervention study ( n = 227).

CDC EZ-Text: Software for Management and Analysis of Semistructured Qualitative Data Sets

Many research projects require collecting semistructured qualitative data. This includes situations where interview guides containing a series of open-ended questions are consistently administered to

Coder training: Theoretical training or practical socialization?

Fragments of a transcript of a coder training are presented that suggest that inter-coder-reliability is improved not only by communicating the coding instructions to coders (theoretical training) but also by socializing coders into practical rules which are not part of the coding Instructions and are not warranted by them.

Experiments in coding clinical information: an analysis of clinicians using a computerized coding tool.

It is concluded that use of a computerized coding tool can save time and result in higher quality coding However, de facto time spent on coding may be underestimated when looking at individual coding times instead of looking at the whole task of processing a clinical scenario.

Can raters consistently evaluate the content of focus groups?

Analysing qualitative interview data: addressing issues of validity and reliability.

The aim of this paper is to describe the steps which the author followed to address issues of rigour in one qualitative research study conducted to explore the health visitor's role in identifying and working with vulnerable families in the community in relation to child protection.

Qualitative research in health care: I. The scope and validity of methods.

Although there is less consensus about appropriate methods of analysing qualitative data, such analyses tend to be grounded in the data, and involve iterative procedures and the development and refinement of typologies, analogies and other forms of concept to make sense of data.

Assessing quality in qualitative research

  • N. MaysC. Pope
  • Sociology, Political Science
    BMJ : British Medical Journal
  • 2000
Two views of how qualitative methods might be judged are outlined and it is argued that qualitative research can be assessed according to two broad criteria: validity and relevance.