Validity of Simulation-Based Assessment for Accreditation Council for Graduate Medical Education Milestone Achievement

  title={Validity of Simulation-Based Assessment for Accreditation Council for Graduate Medical Education Milestone Achievement},
  author={Robert Isaak and Fei Chen and Susan M. Martinelli and Harendra Arora and David A. Zvara and Gene W Hobbs and Marjorie Podraza Stiegler},
  journal={Simulation in Healthcare: The Journal of the Society for Simulation in Healthcare},
  • Robert Isaak, Fei Chen, M. Stiegler
  • Published 1 June 2018
  • Medicine
  • Simulation in Healthcare: The Journal of the Society for Simulation in Healthcare
Introduction The Accreditation Council for Graduate Medical Education requires biannual evaluation of anesthesiology residents on 25 subcompetency milestones. Some milestone domains are particularly challenging to repeatedly and reliably observe during clinical care. Simulation-Based Milestones Assessment (SBMA) may help overcome these challenges. However, few studies have examined the external validation of simulation assessment scores (ie, the relationships between simulation-based assessment… 

Simulation-based evaluation of anaesthesia residents: optimising resource use in a competency-based assessment framework

Simulation-based assessment is becoming an increasingly important tool for assessing the competency of medical residents in conjunction with other assessment methods and G-analysis can be used to assist in planning for optimal resource use and cost-efficacy.

Comparing Real-time Versus Delayed Video Assessments for Evaluating ACGME Sub-competency Milestones in Simulated Patient Care Environments

The findings support the use of either real-time or delayed video evaluation for assessing milestones in a simulated patient care environment, as both assessment methods yielded comparable scores.

Simulation-Based Assessments and Graduating Neurology Residents' Milestones: Status Epilepticus Milestones.

Simulated SE skills performance of graduating neurology residents was poor and this study suggests that end-of-rotation evaluations alone are inadequate for assigning Milestones for high-stakes clinical skills such as identification and management of SE.

Simulation versus real-world performance: a direct comparison of emergency medicine resident resuscitation entrustment scoring

Assessment of residents using entrustment scoring in simulation settings was demonstrated to have a moderate positive correlation with assessment of resuscitation competence in the workplace, suggesting that resuscitation performance in simulation setting may be an indicator of competence inThe clinical setting.

One (Not So Small) Step for Simulation-Based Competency Assessment in Critical Care.

The strength of the study by Boyle et al (4) lies in the rigorous systematic approach to development, which can serve as an example of evidence-based design and evaluation methodology for simulation-based assessment for frontline providers.

Experience Is the Teacher of All Things: Prior Participation in Anesthesiology OSCEs Enhances Communication of Treatment Options With Simulated High-Risk Patients.

Findings from this study provide preliminary evidence that anesthesiology-based OSCE experience is associated with improved performance in an OSCE scenario, specifically regarding discussions of treatment options with high-risk patients.

Creation of Simulation-Based Curriculum of Perioperative Emergencies for Residents in Anesthesiology

A simulation-based curriculum of perioperative emergencies for anesthesiology residents is feasible to implement, viewed as worthwhile by trainees, and can foster education in a different learning environment.

Modules for the Technical Skills Section of the OSCE Component of the American Board of Anesthesiology APPLIED Examination

Video modules and a curriculum for anesthesiology resident OSCE preparation for the Interpretation of Monitors andinterpretation of Echocardiography components and more than 90% of participants reported that the modules were useful, and over 70% reported that they now felt prepared for these stations of the OSCE.

A Scoping Review of Medical Laboratory Science and Simulation: Promoting a Path Forward with Best Practices.

Despite increases in medical laboratory science with simulation, there is a need for more detailed empirical studies, more studies with an interprofessional context, and more methodological rigor.



Standardized Mixed-Fidelity Simulation for ACGME Milestones Competency Assessment and Objective Structured Clinical Exam Preparation

A full-scale, standardized, mixed-fidelity simulation training program for residents that serves multiple functions including the following: efficient and reliable means to meet the bi-annual ACGME reporting standards for milestone competency assessments.

External Validation of Simulation-Based Assessments With Other Performance Measures of Third-Year Anesthesiology Residents

It is hypothesized that technical and nontechnical performance in the simulated environment is related to other various criterion measures, providing evidence to support the validity of the scores from the performance-based assessment.

Simulation-based Assessment to Identify Critical Gaps in Safe Anesthesia Resident Performance

Evidence supporting the reliability and validity of the assessment scores was procured, including a high generalizability coefficient (&rgr;2 = 0.81) and expected performance differences between first-year resident and fellow participants.

Simulated and standardized patients in OSCEs: achievements and challenges 1992-2003

The author discusses achievements and challenges in the use of the objective structured clinical examination (OSCE) and simulated patient (SP) for medical education and assessment and the founding of the Association of Standardized Patient Educators (ASPE), a specialist professional organization.

Mapping Direct Observations From Objective Structured Clinical Examinations to the Milestones Across Specialties.

The ICS-OSCE scores for PGY-1 residents appear to relate to beginner-level milestones for physician-patient communication across multiple specialties.

Barriers to use of simulation-based education

PurposeBarriers to simulation-based education in postgraduate and continuing education for anesthesiologists have not been well studied. We hypothesized that the level of training may influence

Designing and Implementing the Objective Structured Clinical Examination in Anesthesiology

The history of OSCE and its current application in medical education and in different medical and surgical specialties are reviewed and the psychometrics of test design and implementation with emphasis on reliability and validity measures as they relate to OSCE are discussed.

Simulation-based assessment of pediatric anesthesia skills.

A multiple-scenario, simulation-based assessment of pediatric perioperative care was designed and administered to residents and fellows and indicated the content was relevant and that raters could reliably score the scenarios.

Assessing the ACGME general competencies: general considerations and assessment methods.

  • S. Swing
  • Education
    Academic emergency medicine : official journal of the Society for Academic Emergency Medicine
  • 2002
Outcome assessment will provide evidence of residency program educational effectiveness and information to guide improvement and the development and implementations of assessment methods appropriate to evaluate the performance of residents in each of the core competencies.

Direct Observation Assessment of Milestones: Problems with Reliability

The Critical Care Direct Observation Tool (CDOT) was created to assess EM residents during resuscitations using a modified Delphi method focused on content, response process, and internal structure validity, but despite strict attention to validity evidence, inter-rater reliability was low.