Reliability, validity, and feasibility of the zwisch scale for the assessment of intraoperative performance

Brian C. George*, Ezra Nathaniel Teitelbaum, Shari Lynn Meyerson, Mary C. Schuller, Debra DaRosa, Emil R. Petrusa, Lucia Catherine Petito, Jonathan Paul Fryer

*Corresponding author for this work

Research output: Contribution to journalArticle

82 Citations (Scopus)

Abstract

Purpose The existing methods for evaluating resident operative performance interrupt the workflow of the attending physician, are resource intensive, and are often completed well after the end of the procedure in question. These limitations lead to low faculty compliance and potential significant recall bias. In this study, we deployed a smartphone-based system, the Procedural Autonomy and Supervisions System, to facilitate assessment of resident performance according to the Zwisch scale with minimal workflow disruption. We aimed to demonstrate that this is a reliable, valid, and feasible method of measuring resident operative autonomy.

Methods Before implementation, general surgery residents and faculty underwent frame-of-reference training to the Zwisch scale. Immediately after any operation in which a resident participated, the system automatically sent a text message prompting the attending physician to rate the resident's level of operative autonomy according to the 4-level Zwisch scale. Of these procedures, 8 were videotaped and independently rated by 2 additional surgeons. The Zwisch ratings of the 3 raters were compared using an intraclass correlation coefficient. Videotaped procedures were also scored using 2 alternative operating room (OR) performance assessment instruments (Operative Performance Rating System and Ottawa Surgical Competency OR Evaluation), against which the item correlations were calculated.

Results Between December 2012 and June 2013, 27 faculty used the smartphone system to complete 1490 operative performance assessments on 31 residents. During this period, faculty completed evaluations for 92% of all operations performed with general surgery residents. The Zwisch scores were shown to correlate with postgraduate year (PGY) levels based on sequential pairwise chi-squared tests: PGY 1 vs PGY 2 (χ2 = 106.9, df = 3, p < 0.001); PGY 2 vs PGY 3 (χ2 = 22.2, df = 3, p < 0.001); and PGY 3 vs PGY 4 (χ2 = 56.4, df = 3, p < 0.001). Comparison of PGY 4 to PGY 5 scores were not significantly different (χ2 = 4.5, df = 3, p = 0.21). For the 8 operations reviewed for interrater reliability, the intraclass correlation coefficient was 0.90 (95% CI: 0.72-0.98, p < 0.01). Correlation of Procedural Autonomy and Supervisions System ratings with both Operative Performance Rating System items (each r > 0.90, all p's < 0.01) and Ottawa Surgical Competency OR Evaluation items (each r > 0.86, all p's < 0.01) was high.

Conclusions The Zwisch scale can be used to make reliable and valid measurements of faculty guidance and resident autonomy. Our data also suggest that Zwisch ratings may be used to infer resident operative performance. Deployed on an automated smartphone-based system, it can be used to feasibly record evaluations for most operations performed by residents. This information can be used to council individual residents, modify programmatic curricula, and potentially inform national training guidelines.

Original languageEnglish (US)
Pages (from-to)e90-e96
JournalJournal of Surgical Education
Volume71
Issue number6
DOIs
StatePublished - Nov 1 2014

Fingerprint

Reproducibility of Results
resident
performance
Workflow
Operating Rooms
autonomy
Text Messaging
Physicians
performance assessment
rating
workflow
Curriculum
surgery
Compliance
evaluation
physician
Guidelines
supervision
Smartphone
curriculum

Keywords

  • educa- tional measurement
  • evaluation
  • graduate medical education
  • surgery
  • surgical education

ASJC Scopus subject areas

  • Surgery
  • Education

Cite this

@article{27e796e8897c47058a6f6ebec8e53d5e,
title = "Reliability, validity, and feasibility of the zwisch scale for the assessment of intraoperative performance",
abstract = "Purpose The existing methods for evaluating resident operative performance interrupt the workflow of the attending physician, are resource intensive, and are often completed well after the end of the procedure in question. These limitations lead to low faculty compliance and potential significant recall bias. In this study, we deployed a smartphone-based system, the Procedural Autonomy and Supervisions System, to facilitate assessment of resident performance according to the Zwisch scale with minimal workflow disruption. We aimed to demonstrate that this is a reliable, valid, and feasible method of measuring resident operative autonomy.Methods Before implementation, general surgery residents and faculty underwent frame-of-reference training to the Zwisch scale. Immediately after any operation in which a resident participated, the system automatically sent a text message prompting the attending physician to rate the resident's level of operative autonomy according to the 4-level Zwisch scale. Of these procedures, 8 were videotaped and independently rated by 2 additional surgeons. The Zwisch ratings of the 3 raters were compared using an intraclass correlation coefficient. Videotaped procedures were also scored using 2 alternative operating room (OR) performance assessment instruments (Operative Performance Rating System and Ottawa Surgical Competency OR Evaluation), against which the item correlations were calculated.Results Between December 2012 and June 2013, 27 faculty used the smartphone system to complete 1490 operative performance assessments on 31 residents. During this period, faculty completed evaluations for 92{\%} of all operations performed with general surgery residents. The Zwisch scores were shown to correlate with postgraduate year (PGY) levels based on sequential pairwise chi-squared tests: PGY 1 vs PGY 2 (χ2 = 106.9, df = 3, p < 0.001); PGY 2 vs PGY 3 (χ2 = 22.2, df = 3, p < 0.001); and PGY 3 vs PGY 4 (χ2 = 56.4, df = 3, p < 0.001). Comparison of PGY 4 to PGY 5 scores were not significantly different (χ2 = 4.5, df = 3, p = 0.21). For the 8 operations reviewed for interrater reliability, the intraclass correlation coefficient was 0.90 (95{\%} CI: 0.72-0.98, p < 0.01). Correlation of Procedural Autonomy and Supervisions System ratings with both Operative Performance Rating System items (each r > 0.90, all p's < 0.01) and Ottawa Surgical Competency OR Evaluation items (each r > 0.86, all p's < 0.01) was high.Conclusions The Zwisch scale can be used to make reliable and valid measurements of faculty guidance and resident autonomy. Our data also suggest that Zwisch ratings may be used to infer resident operative performance. Deployed on an automated smartphone-based system, it can be used to feasibly record evaluations for most operations performed by residents. This information can be used to council individual residents, modify programmatic curricula, and potentially inform national training guidelines.",
keywords = "educa- tional measurement, evaluation, graduate medical education, surgery, surgical education",
author = "George, {Brian C.} and Teitelbaum, {Ezra Nathaniel} and Meyerson, {Shari Lynn} and Schuller, {Mary C.} and Debra DaRosa and Petrusa, {Emil R.} and Petito, {Lucia Catherine} and Fryer, {Jonathan Paul}",
year = "2014",
month = "11",
day = "1",
doi = "10.1016/j.jsurg.2014.06.018",
language = "English (US)",
volume = "71",
pages = "e90--e96",
journal = "Journal of Surgical Education",
issn = "1931-7204",
publisher = "Elsevier Inc.",
number = "6",

}

Reliability, validity, and feasibility of the zwisch scale for the assessment of intraoperative performance. / George, Brian C.; Teitelbaum, Ezra Nathaniel; Meyerson, Shari Lynn; Schuller, Mary C.; DaRosa, Debra; Petrusa, Emil R.; Petito, Lucia Catherine; Fryer, Jonathan Paul.

In: Journal of Surgical Education, Vol. 71, No. 6, 01.11.2014, p. e90-e96.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Reliability, validity, and feasibility of the zwisch scale for the assessment of intraoperative performance

AU - George, Brian C.

AU - Teitelbaum, Ezra Nathaniel

AU - Meyerson, Shari Lynn

AU - Schuller, Mary C.

AU - DaRosa, Debra

AU - Petrusa, Emil R.

AU - Petito, Lucia Catherine

AU - Fryer, Jonathan Paul

PY - 2014/11/1

Y1 - 2014/11/1

N2 - Purpose The existing methods for evaluating resident operative performance interrupt the workflow of the attending physician, are resource intensive, and are often completed well after the end of the procedure in question. These limitations lead to low faculty compliance and potential significant recall bias. In this study, we deployed a smartphone-based system, the Procedural Autonomy and Supervisions System, to facilitate assessment of resident performance according to the Zwisch scale with minimal workflow disruption. We aimed to demonstrate that this is a reliable, valid, and feasible method of measuring resident operative autonomy.Methods Before implementation, general surgery residents and faculty underwent frame-of-reference training to the Zwisch scale. Immediately after any operation in which a resident participated, the system automatically sent a text message prompting the attending physician to rate the resident's level of operative autonomy according to the 4-level Zwisch scale. Of these procedures, 8 were videotaped and independently rated by 2 additional surgeons. The Zwisch ratings of the 3 raters were compared using an intraclass correlation coefficient. Videotaped procedures were also scored using 2 alternative operating room (OR) performance assessment instruments (Operative Performance Rating System and Ottawa Surgical Competency OR Evaluation), against which the item correlations were calculated.Results Between December 2012 and June 2013, 27 faculty used the smartphone system to complete 1490 operative performance assessments on 31 residents. During this period, faculty completed evaluations for 92% of all operations performed with general surgery residents. The Zwisch scores were shown to correlate with postgraduate year (PGY) levels based on sequential pairwise chi-squared tests: PGY 1 vs PGY 2 (χ2 = 106.9, df = 3, p < 0.001); PGY 2 vs PGY 3 (χ2 = 22.2, df = 3, p < 0.001); and PGY 3 vs PGY 4 (χ2 = 56.4, df = 3, p < 0.001). Comparison of PGY 4 to PGY 5 scores were not significantly different (χ2 = 4.5, df = 3, p = 0.21). For the 8 operations reviewed for interrater reliability, the intraclass correlation coefficient was 0.90 (95% CI: 0.72-0.98, p < 0.01). Correlation of Procedural Autonomy and Supervisions System ratings with both Operative Performance Rating System items (each r > 0.90, all p's < 0.01) and Ottawa Surgical Competency OR Evaluation items (each r > 0.86, all p's < 0.01) was high.Conclusions The Zwisch scale can be used to make reliable and valid measurements of faculty guidance and resident autonomy. Our data also suggest that Zwisch ratings may be used to infer resident operative performance. Deployed on an automated smartphone-based system, it can be used to feasibly record evaluations for most operations performed by residents. This information can be used to council individual residents, modify programmatic curricula, and potentially inform national training guidelines.

AB - Purpose The existing methods for evaluating resident operative performance interrupt the workflow of the attending physician, are resource intensive, and are often completed well after the end of the procedure in question. These limitations lead to low faculty compliance and potential significant recall bias. In this study, we deployed a smartphone-based system, the Procedural Autonomy and Supervisions System, to facilitate assessment of resident performance according to the Zwisch scale with minimal workflow disruption. We aimed to demonstrate that this is a reliable, valid, and feasible method of measuring resident operative autonomy.Methods Before implementation, general surgery residents and faculty underwent frame-of-reference training to the Zwisch scale. Immediately after any operation in which a resident participated, the system automatically sent a text message prompting the attending physician to rate the resident's level of operative autonomy according to the 4-level Zwisch scale. Of these procedures, 8 were videotaped and independently rated by 2 additional surgeons. The Zwisch ratings of the 3 raters were compared using an intraclass correlation coefficient. Videotaped procedures were also scored using 2 alternative operating room (OR) performance assessment instruments (Operative Performance Rating System and Ottawa Surgical Competency OR Evaluation), against which the item correlations were calculated.Results Between December 2012 and June 2013, 27 faculty used the smartphone system to complete 1490 operative performance assessments on 31 residents. During this period, faculty completed evaluations for 92% of all operations performed with general surgery residents. The Zwisch scores were shown to correlate with postgraduate year (PGY) levels based on sequential pairwise chi-squared tests: PGY 1 vs PGY 2 (χ2 = 106.9, df = 3, p < 0.001); PGY 2 vs PGY 3 (χ2 = 22.2, df = 3, p < 0.001); and PGY 3 vs PGY 4 (χ2 = 56.4, df = 3, p < 0.001). Comparison of PGY 4 to PGY 5 scores were not significantly different (χ2 = 4.5, df = 3, p = 0.21). For the 8 operations reviewed for interrater reliability, the intraclass correlation coefficient was 0.90 (95% CI: 0.72-0.98, p < 0.01). Correlation of Procedural Autonomy and Supervisions System ratings with both Operative Performance Rating System items (each r > 0.90, all p's < 0.01) and Ottawa Surgical Competency OR Evaluation items (each r > 0.86, all p's < 0.01) was high.Conclusions The Zwisch scale can be used to make reliable and valid measurements of faculty guidance and resident autonomy. Our data also suggest that Zwisch ratings may be used to infer resident operative performance. Deployed on an automated smartphone-based system, it can be used to feasibly record evaluations for most operations performed by residents. This information can be used to council individual residents, modify programmatic curricula, and potentially inform national training guidelines.

KW - educa- tional measurement

KW - evaluation

KW - graduate medical education

KW - surgery

KW - surgical education

UR - http://www.scopus.com/inward/record.url?scp=84919956693&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84919956693&partnerID=8YFLogxK

U2 - 10.1016/j.jsurg.2014.06.018

DO - 10.1016/j.jsurg.2014.06.018

M3 - Article

VL - 71

SP - e90-e96

JO - Journal of Surgical Education

JF - Journal of Surgical Education

SN - 1931-7204

IS - 6

ER -