How Many Observations are Needed to Assess a Surgical Trainee's State of Operative Competency?

Reed G. Williams, David B. Swanson, Jonathan Paul Fryer, Shari Lynn Meyerson, Jordan D. Bohnen, Gary L. Dunnington, Rebecca E. Scully, Mary C. Schuller, Brian C. George

Research output: Contribution to journalArticle

2 Citations (Scopus)

Abstract

OBJECTIVE: To establish the number of operative performance observations needed for reproducible assessments of operative competency. BACKGROUND: Surgical training is transitioning from a time-based to a competency-based approach, but the number of assessments needed to reliably establish operative competency remains unknown. METHODS: Using a smart phone based operative evaluation application (SIMPL), residents from 13 general surgery training programs were evaluated performing common surgical procedures. Two competency metrics were investigated separately: autonomy and overall performance. Analyses were performed for laparoscopic cholecystectomy performances alone and for all operative procedures combined. Variance component analyses determined operative performance score variance attributable to resident operative competency and measurement error. Generalizability and decision studies determined number of assessments needed to achieve desired reliability (0.80 or greater) and determine standard errors of measurement. RESULTS: For laparoscopic cholecystectomy, 23 ratings are needed to achieve reproducible autonomy ratings and 17 ratings are needed to achieve reproducible overall operative performance ratings. For the undifferentiated mix of procedures, 60 ratings are needed to achieve reproducible autonomy ratings and 40 are needed for reproducible overall operative performance ratings. CONCLUSION: The number of observations needed to achieve reproducible assessments of operative competency far exceeds current certification requirements, yet remains an important and achievable goal. Attention should also be paid to the mix of cases and raters in order to assure fair judgments about operative competency and fair comparisons of trainees.

Original languageEnglish (US)
Pages (from-to)377-382
Number of pages6
JournalAnnals of Surgery
Volume269
Issue number2
DOIs
StatePublished - Feb 1 2019

Fingerprint

Laparoscopic Cholecystectomy
Diagnosis-Related Groups
Operative Surgical Procedures
Certification
Analysis of Variance
Education
Smartphone

ASJC Scopus subject areas

  • Surgery

Cite this

Williams, R. G., Swanson, D. B., Fryer, J. P., Meyerson, S. L., Bohnen, J. D., Dunnington, G. L., ... George, B. C. (2019). How Many Observations are Needed to Assess a Surgical Trainee's State of Operative Competency? Annals of Surgery, 269(2), 377-382. https://doi.org/10.1097/SLA.0000000000002554
Williams, Reed G. ; Swanson, David B. ; Fryer, Jonathan Paul ; Meyerson, Shari Lynn ; Bohnen, Jordan D. ; Dunnington, Gary L. ; Scully, Rebecca E. ; Schuller, Mary C. ; George, Brian C. / How Many Observations are Needed to Assess a Surgical Trainee's State of Operative Competency?. In: Annals of Surgery. 2019 ; Vol. 269, No. 2. pp. 377-382.
@article{a0aef81932514da6918fabefd66f7b41,
title = "How Many Observations are Needed to Assess a Surgical Trainee's State of Operative Competency?",
abstract = "OBJECTIVE: To establish the number of operative performance observations needed for reproducible assessments of operative competency. BACKGROUND: Surgical training is transitioning from a time-based to a competency-based approach, but the number of assessments needed to reliably establish operative competency remains unknown. METHODS: Using a smart phone based operative evaluation application (SIMPL), residents from 13 general surgery training programs were evaluated performing common surgical procedures. Two competency metrics were investigated separately: autonomy and overall performance. Analyses were performed for laparoscopic cholecystectomy performances alone and for all operative procedures combined. Variance component analyses determined operative performance score variance attributable to resident operative competency and measurement error. Generalizability and decision studies determined number of assessments needed to achieve desired reliability (0.80 or greater) and determine standard errors of measurement. RESULTS: For laparoscopic cholecystectomy, 23 ratings are needed to achieve reproducible autonomy ratings and 17 ratings are needed to achieve reproducible overall operative performance ratings. For the undifferentiated mix of procedures, 60 ratings are needed to achieve reproducible autonomy ratings and 40 are needed for reproducible overall operative performance ratings. CONCLUSION: The number of observations needed to achieve reproducible assessments of operative competency far exceeds current certification requirements, yet remains an important and achievable goal. Attention should also be paid to the mix of cases and raters in order to assure fair judgments about operative competency and fair comparisons of trainees.",
author = "Williams, {Reed G.} and Swanson, {David B.} and Fryer, {Jonathan Paul} and Meyerson, {Shari Lynn} and Bohnen, {Jordan D.} and Dunnington, {Gary L.} and Scully, {Rebecca E.} and Schuller, {Mary C.} and George, {Brian C.}",
year = "2019",
month = "2",
day = "1",
doi = "10.1097/SLA.0000000000002554",
language = "English (US)",
volume = "269",
pages = "377--382",
journal = "Annals of Surgery",
issn = "0003-4932",
publisher = "Lippincott Williams and Wilkins",
number = "2",

}

Williams, RG, Swanson, DB, Fryer, JP, Meyerson, SL, Bohnen, JD, Dunnington, GL, Scully, RE, Schuller, MC & George, BC 2019, 'How Many Observations are Needed to Assess a Surgical Trainee's State of Operative Competency?', Annals of Surgery, vol. 269, no. 2, pp. 377-382. https://doi.org/10.1097/SLA.0000000000002554

How Many Observations are Needed to Assess a Surgical Trainee's State of Operative Competency? / Williams, Reed G.; Swanson, David B.; Fryer, Jonathan Paul; Meyerson, Shari Lynn; Bohnen, Jordan D.; Dunnington, Gary L.; Scully, Rebecca E.; Schuller, Mary C.; George, Brian C.

In: Annals of Surgery, Vol. 269, No. 2, 01.02.2019, p. 377-382.

Research output: Contribution to journalArticle

TY - JOUR

T1 - How Many Observations are Needed to Assess a Surgical Trainee's State of Operative Competency?

AU - Williams, Reed G.

AU - Swanson, David B.

AU - Fryer, Jonathan Paul

AU - Meyerson, Shari Lynn

AU - Bohnen, Jordan D.

AU - Dunnington, Gary L.

AU - Scully, Rebecca E.

AU - Schuller, Mary C.

AU - George, Brian C.

PY - 2019/2/1

Y1 - 2019/2/1

N2 - OBJECTIVE: To establish the number of operative performance observations needed for reproducible assessments of operative competency. BACKGROUND: Surgical training is transitioning from a time-based to a competency-based approach, but the number of assessments needed to reliably establish operative competency remains unknown. METHODS: Using a smart phone based operative evaluation application (SIMPL), residents from 13 general surgery training programs were evaluated performing common surgical procedures. Two competency metrics were investigated separately: autonomy and overall performance. Analyses were performed for laparoscopic cholecystectomy performances alone and for all operative procedures combined. Variance component analyses determined operative performance score variance attributable to resident operative competency and measurement error. Generalizability and decision studies determined number of assessments needed to achieve desired reliability (0.80 or greater) and determine standard errors of measurement. RESULTS: For laparoscopic cholecystectomy, 23 ratings are needed to achieve reproducible autonomy ratings and 17 ratings are needed to achieve reproducible overall operative performance ratings. For the undifferentiated mix of procedures, 60 ratings are needed to achieve reproducible autonomy ratings and 40 are needed for reproducible overall operative performance ratings. CONCLUSION: The number of observations needed to achieve reproducible assessments of operative competency far exceeds current certification requirements, yet remains an important and achievable goal. Attention should also be paid to the mix of cases and raters in order to assure fair judgments about operative competency and fair comparisons of trainees.

AB - OBJECTIVE: To establish the number of operative performance observations needed for reproducible assessments of operative competency. BACKGROUND: Surgical training is transitioning from a time-based to a competency-based approach, but the number of assessments needed to reliably establish operative competency remains unknown. METHODS: Using a smart phone based operative evaluation application (SIMPL), residents from 13 general surgery training programs were evaluated performing common surgical procedures. Two competency metrics were investigated separately: autonomy and overall performance. Analyses were performed for laparoscopic cholecystectomy performances alone and for all operative procedures combined. Variance component analyses determined operative performance score variance attributable to resident operative competency and measurement error. Generalizability and decision studies determined number of assessments needed to achieve desired reliability (0.80 or greater) and determine standard errors of measurement. RESULTS: For laparoscopic cholecystectomy, 23 ratings are needed to achieve reproducible autonomy ratings and 17 ratings are needed to achieve reproducible overall operative performance ratings. For the undifferentiated mix of procedures, 60 ratings are needed to achieve reproducible autonomy ratings and 40 are needed for reproducible overall operative performance ratings. CONCLUSION: The number of observations needed to achieve reproducible assessments of operative competency far exceeds current certification requirements, yet remains an important and achievable goal. Attention should also be paid to the mix of cases and raters in order to assure fair judgments about operative competency and fair comparisons of trainees.

UR - http://www.scopus.com/inward/record.url?scp=85053618587&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85053618587&partnerID=8YFLogxK

U2 - 10.1097/SLA.0000000000002554

DO - 10.1097/SLA.0000000000002554

M3 - Article

VL - 269

SP - 377

EP - 382

JO - Annals of Surgery

JF - Annals of Surgery

SN - 0003-4932

IS - 2

ER -