Improving resident performance assessment data: Numeric precision and narrative specificity

John H. Littlefield*, Debra A. DaRosa, Judy Paukert, Reed G. Williams, Debra L. Klamen, John D. Schoolfield

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

19 Scopus citations

Abstract

Purpose: To evaluate the use of a systems approach for diagnosing performance assessment problems in surgery residencies, and intervene to improve the numeric precision of global rating scores and the behavioral specificity of narrative comments. Method: Faculty and residents at two surgery programs participated in parallel before-and-after trials. During the baseline year, quality assurance data were gathered and problems were identified. During two subsequent intervention years, an educational specialist at each program intervened with an organizational change strategy to improve information feedback loops. Three quality-assurance measures were analyzed: (1) percentage return rate of forms, (2) generalizability coefficients and 95% confidence intervals of scores, and (3) percentage of forms with behaviorally specific narrative comments. Results: Median return rates of forms increased significantly from baseline to intervention Year 1 at Site A (71% to 100%) and Site B (75% to 100%), and then remained stable during Year 2. Generalizability coefficients increased between baseline and intervention Year 1 at Site A (0.65 to 0.85) and Site B (0.58 to 0.79), and then remained stable. The 95% confidence interval around resident mean scores improved at Site A from baseline to intervention Year 1 (0.78 to 0.58) and then remained stable; at Site B, it remained constant throughout (0.55 to 0.56). The median percentage of forms with behaviorally specific narrative comments at Site A increased significantly from baseline to intervention Years 1 and 2 (50%, 57%, 82%); at Site B, the percentage increased significantly in intervention Year 1, and then remained constant (50%, 60%, 67%). Conclusions: Diagnosing performance assessment system problems and improving information feedback loops improved the quality of resident performance assessment data at both programs.

Original languageEnglish (US)
Pages (from-to)489-495
Number of pages7
JournalAcademic Medicine
Volume80
Issue number5
DOIs
StatePublished - May 2005

ASJC Scopus subject areas

  • Education

Fingerprint

Dive into the research topics of 'Improving resident performance assessment data: Numeric precision and narrative specificity'. Together they form a unique fingerprint.

Cite this