Evaluating Surgical Coaching: A Mixed Methods Approach Reveals More Than Surveys Alone

Objective: Traditionally, surgical educators have relied upon participant survey data for the evaluation of educational interventions. However, the ability of such subjective data to completely evaluate an intervention is limited. Our objective was to compare resident and attending surgeons’ self-assessments of coaching sessions from surveys with independent observations from analysis of intraoperative and postoperative coaching transcripts. Design: Senior residents were video-recorded operating. Each was then coached by the operative attending in a 1:1 video review session. Teaching points made in the operating room (OR) and in post-OR coaching sessions were coded by independent observers using dialogue analysis then compared using t-tests. Participants were surveyed regarding the degree of teaching dedicated to specific topics and perceived changes in teaching level, resident comfort, educational assessments, and feedback provision between the OR and the post-OR coaching sessions. Setting: A single, large, urban, tertiary-care academic institution. Participants: Ten PGY4 to 5 general surgery residents and 10 attending surgeons. Results: Although the reported experiences of teaching and coaching sessions by residents and faculty were similar (Pearson correlation coefficient = 0.88), these differed significantly from independent observations. Observers found that residents initiated a greater proportion of teaching points and had more educational needs assessments during coaching, compared to the OR. However, neither residents nor attendings reported a change between the 2 environments with regard to needs assessments nor comfort with asking questions or making suggestions. The only metric on which residents, attendings, and observers agreed was the provision of feedback. Conclusions: Participants’ perspectives, although considered highly reliable by traditional metrics, rarely aligned with analysis of the associated transcripts from independent observers. Independent observation showed a distinct benefit of coaching in terms of frequency and type of learning points. These findings highlight the importance of seeking different perspectives, data sources, and methodologies when evaluating clinical education interventions. Surgical education can benefit from increased use of dialogue analyses performed by independent observers, which may represent a viewpoint distinct from that obtained by survey methodology.