Duration of faculty training needed to ensure reliable or performance ratings

Brian C. George*, Ezra Nathaniel Teitelbaum, Debra DaRosa, Eric Hungness, Shari Lynn Meyerson, Jonathan Paul Fryer, Mary Schuller, Joseph B. Zwischenberger

*Corresponding author for this work

Research output: Contribution to journalArticle

22 Citations (Scopus)

Abstract

OBJECTIVES: The American Board of Surgery has mandated intraoperative assessment of general surgery residents, yet the time required to train faculty to accurately and reliably complete operating room performance evaluation forms is unknown. Outside of surgical education, frame-of-reference (FOR) training has been shown to be an effective training modality to teach raters the specific performance indicators associated with each point on a rating scale. Little is known, however, about what form and duration of FOR training is needed to accomplish reliable ratings among surgical faculty. DESIGN: Two groups of surgical faculty separately underwent either an accelerated 1-hour (n = 10) or immersive four-hour (n = 34) FOR faculty development program. Both programs included a formal presentation and a facilitated discussion of sample behaviors for each point on the Zwisch operating room performance rating scale (see DaRosa et al.8). The immersive group additionally participated in a small group exercise that included additional practice. After training, both groups were tested using 10 video clips of trainees at various levels. Responses were scored against expert consensus ratings. The 2-sided Mann-Whitney U test was used to compare between group means. SETTING AND PARTICIPANTS: All trainees were faculty members in the Department of Surgery of a large midwestern private medical school. RESULTS: Faculty undergoing the 1-hour FOR training program did not have a statistically different mean correct response rate on the video test when compared with those undergoing the 4-hour training program (88% vs 80%; p = 0.07). CONCLUSIONS: One-hour FOR training sessions are likely sufficient to train surgical faculty to reliably use a simple evaluation instrument for the assessment of intra-operative performance. Additional research is needed to determine how these results generalize to different assessment instruments. (J Surg 70:703-708.

Original languageEnglish (US)
Pages (from-to)703-708
Number of pages6
JournalJournal of Surgical Education
Volume70
Issue number6
DOIs
StatePublished - Jan 1 2013

Fingerprint

rating
performance
surgery
Operating Rooms
Education
rating scale
trainee
training program
Group
video clip
Nonparametric Statistics
Medical Schools
Surgical Instruments
evaluation
small group
Exercise
video
expert
resident
Research

Keywords

  • Evaluation
  • Faculty development
  • Frame-of-reference training
  • Measurement
  • Rater training
  • Surgical education

ASJC Scopus subject areas

  • Surgery
  • Education

Cite this

George, Brian C. ; Teitelbaum, Ezra Nathaniel ; DaRosa, Debra ; Hungness, Eric ; Meyerson, Shari Lynn ; Fryer, Jonathan Paul ; Schuller, Mary ; Zwischenberger, Joseph B. / Duration of faculty training needed to ensure reliable or performance ratings. In: Journal of Surgical Education. 2013 ; Vol. 70, No. 6. pp. 703-708.
@article{3133dafae6f04130b64c92c3d4500d9e,
title = "Duration of faculty training needed to ensure reliable or performance ratings",
abstract = "OBJECTIVES: The American Board of Surgery has mandated intraoperative assessment of general surgery residents, yet the time required to train faculty to accurately and reliably complete operating room performance evaluation forms is unknown. Outside of surgical education, frame-of-reference (FOR) training has been shown to be an effective training modality to teach raters the specific performance indicators associated with each point on a rating scale. Little is known, however, about what form and duration of FOR training is needed to accomplish reliable ratings among surgical faculty. DESIGN: Two groups of surgical faculty separately underwent either an accelerated 1-hour (n = 10) or immersive four-hour (n = 34) FOR faculty development program. Both programs included a formal presentation and a facilitated discussion of sample behaviors for each point on the Zwisch operating room performance rating scale (see DaRosa et al.8). The immersive group additionally participated in a small group exercise that included additional practice. After training, both groups were tested using 10 video clips of trainees at various levels. Responses were scored against expert consensus ratings. The 2-sided Mann-Whitney U test was used to compare between group means. SETTING AND PARTICIPANTS: All trainees were faculty members in the Department of Surgery of a large midwestern private medical school. RESULTS: Faculty undergoing the 1-hour FOR training program did not have a statistically different mean correct response rate on the video test when compared with those undergoing the 4-hour training program (88{\%} vs 80{\%}; p = 0.07). CONCLUSIONS: One-hour FOR training sessions are likely sufficient to train surgical faculty to reliably use a simple evaluation instrument for the assessment of intra-operative performance. Additional research is needed to determine how these results generalize to different assessment instruments. (J Surg 70:703-708.",
keywords = "Evaluation, Faculty development, Frame-of-reference training, Measurement, Rater training, Surgical education",
author = "George, {Brian C.} and Teitelbaum, {Ezra Nathaniel} and Debra DaRosa and Eric Hungness and Meyerson, {Shari Lynn} and Fryer, {Jonathan Paul} and Mary Schuller and Zwischenberger, {Joseph B.}",
year = "2013",
month = "1",
day = "1",
doi = "10.1016/j.jsurg.2013.06.015",
language = "English (US)",
volume = "70",
pages = "703--708",
journal = "Journal of Surgical Education",
issn = "1931-7204",
publisher = "Elsevier Inc.",
number = "6",

}

Duration of faculty training needed to ensure reliable or performance ratings. / George, Brian C.; Teitelbaum, Ezra Nathaniel; DaRosa, Debra; Hungness, Eric; Meyerson, Shari Lynn; Fryer, Jonathan Paul; Schuller, Mary; Zwischenberger, Joseph B.

In: Journal of Surgical Education, Vol. 70, No. 6, 01.01.2013, p. 703-708.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Duration of faculty training needed to ensure reliable or performance ratings

AU - George, Brian C.

AU - Teitelbaum, Ezra Nathaniel

AU - DaRosa, Debra

AU - Hungness, Eric

AU - Meyerson, Shari Lynn

AU - Fryer, Jonathan Paul

AU - Schuller, Mary

AU - Zwischenberger, Joseph B.

PY - 2013/1/1

Y1 - 2013/1/1

N2 - OBJECTIVES: The American Board of Surgery has mandated intraoperative assessment of general surgery residents, yet the time required to train faculty to accurately and reliably complete operating room performance evaluation forms is unknown. Outside of surgical education, frame-of-reference (FOR) training has been shown to be an effective training modality to teach raters the specific performance indicators associated with each point on a rating scale. Little is known, however, about what form and duration of FOR training is needed to accomplish reliable ratings among surgical faculty. DESIGN: Two groups of surgical faculty separately underwent either an accelerated 1-hour (n = 10) or immersive four-hour (n = 34) FOR faculty development program. Both programs included a formal presentation and a facilitated discussion of sample behaviors for each point on the Zwisch operating room performance rating scale (see DaRosa et al.8). The immersive group additionally participated in a small group exercise that included additional practice. After training, both groups were tested using 10 video clips of trainees at various levels. Responses were scored against expert consensus ratings. The 2-sided Mann-Whitney U test was used to compare between group means. SETTING AND PARTICIPANTS: All trainees were faculty members in the Department of Surgery of a large midwestern private medical school. RESULTS: Faculty undergoing the 1-hour FOR training program did not have a statistically different mean correct response rate on the video test when compared with those undergoing the 4-hour training program (88% vs 80%; p = 0.07). CONCLUSIONS: One-hour FOR training sessions are likely sufficient to train surgical faculty to reliably use a simple evaluation instrument for the assessment of intra-operative performance. Additional research is needed to determine how these results generalize to different assessment instruments. (J Surg 70:703-708.

AB - OBJECTIVES: The American Board of Surgery has mandated intraoperative assessment of general surgery residents, yet the time required to train faculty to accurately and reliably complete operating room performance evaluation forms is unknown. Outside of surgical education, frame-of-reference (FOR) training has been shown to be an effective training modality to teach raters the specific performance indicators associated with each point on a rating scale. Little is known, however, about what form and duration of FOR training is needed to accomplish reliable ratings among surgical faculty. DESIGN: Two groups of surgical faculty separately underwent either an accelerated 1-hour (n = 10) or immersive four-hour (n = 34) FOR faculty development program. Both programs included a formal presentation and a facilitated discussion of sample behaviors for each point on the Zwisch operating room performance rating scale (see DaRosa et al.8). The immersive group additionally participated in a small group exercise that included additional practice. After training, both groups were tested using 10 video clips of trainees at various levels. Responses were scored against expert consensus ratings. The 2-sided Mann-Whitney U test was used to compare between group means. SETTING AND PARTICIPANTS: All trainees were faculty members in the Department of Surgery of a large midwestern private medical school. RESULTS: Faculty undergoing the 1-hour FOR training program did not have a statistically different mean correct response rate on the video test when compared with those undergoing the 4-hour training program (88% vs 80%; p = 0.07). CONCLUSIONS: One-hour FOR training sessions are likely sufficient to train surgical faculty to reliably use a simple evaluation instrument for the assessment of intra-operative performance. Additional research is needed to determine how these results generalize to different assessment instruments. (J Surg 70:703-708.

KW - Evaluation

KW - Faculty development

KW - Frame-of-reference training

KW - Measurement

KW - Rater training

KW - Surgical education

UR - http://www.scopus.com/inward/record.url?scp=84892787989&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84892787989&partnerID=8YFLogxK

U2 - 10.1016/j.jsurg.2013.06.015

DO - 10.1016/j.jsurg.2013.06.015

M3 - Article

VL - 70

SP - 703

EP - 708

JO - Journal of Surgical Education

JF - Journal of Surgical Education

SN - 1931-7204

IS - 6

ER -