Resident Surgical Skills Web-Based Evaluation: A Comparison of 2 Assessment Tools

Ann E. Van Heest, Julie Agel, S. Elizabeth Ames, Ferhan A. Asghar, John J. Harrast, J. Lawrence Marsh, Joshua C. Patt, Robert S. Sterling, Terrance D Peabody

Research output: Contribution to journalArticle

Abstract

Background:Evaluation of surgical skill competency is necessary as graduate medical education moves toward a competency-based curriculum. This study by the American Board of Orthopaedic Surgery (ABOS) and the Council of Orthopaedic Residency Directors (CORD) compares 2 web-based evaluation tools that assess the level of autonomy that is demonstrated by residents during surgical procedures in the operating room as measured by faculty.Methods:Two hundred and ninety-four residents from 16 orthopaedic surgery residency programs were evaluated by 370 faculty using 2 web-based evaluation tools in a crossover design in which residents requested faculty review of their surgical skills before starting a case. One thousand, one hundred and fifty Ottawa Surgical Competency Operating Room Evaluation (O-Score) assessments, which included a 9-question evaluation of 8 steps of the surgical procedure, were compared with 1,186 P-score evaluations, which included a single-question summative evaluation. Twenty-five different surgical procedures were evaluated.Results:There were no significant differences in rates of resident requests or faculty completion of the 2 scores. The most common surgical procedures that were assessed were total knee arthroplasty (n = 254, 11%), carpal tunnel release (n = 191, 8%), open reduction and internal fixation (ORIF) of stable hip fractures (n = 170, 7%), ORIF of simple ankle fractures (n = 169, 7%), and total hip arthroplasty (n = 166, 7%). Both instruments disclosed significant differences in competency among entry, intermediate, and advanced-level residents. The findings support the construct validity of the evaluation method. The survey results indicated that >70% of the faculty were confident that use of either the P-score or the O-score allowed them to distinguish a resident who can perform the surgery independently from one who needs additional training.Conclusions:This research has led to the modification of the O-score and the P-score into a combined OP-score instrument. The ABOS envisions that the OP-score instrument can be used with an expanded number of surgical procedures as a required element of residency training in the near future.Clinical Relevance:This study allows the profession of orthopaedic surgery education to take a leadership role in the measurement of competence for surgical skills for orthopaedic surgeons in residency training, an important clinically relevant topic to the practice of orthopaedic surgery.

Original languageEnglish (US)
Article numbere18
JournalJournal of Bone and Joint Surgery - American Volume
Volume101
Issue number5
DOIs
StatePublished - Mar 6 2019

Fingerprint

Orthopedics
Internship and Residency
Operating Rooms
Ankle Fractures
Graduate Medical Education
Knee Replacement Arthroplasties
Hip Fractures
Wrist
Arthroplasty
Curriculum
Cross-Over Studies
Mental Competency
Hip
Education
Research

ASJC Scopus subject areas

  • Surgery
  • Orthopedics and Sports Medicine

Cite this

Van Heest, Ann E. ; Agel, Julie ; Ames, S. Elizabeth ; Asghar, Ferhan A. ; Harrast, John J. ; Marsh, J. Lawrence ; Patt, Joshua C. ; Sterling, Robert S. ; Peabody, Terrance D. / Resident Surgical Skills Web-Based Evaluation : A Comparison of 2 Assessment Tools. In: Journal of Bone and Joint Surgery - American Volume. 2019 ; Vol. 101, No. 5.
@article{0e3d42774eec467a8aa260eeb6ed9c4c,
title = "Resident Surgical Skills Web-Based Evaluation: A Comparison of 2 Assessment Tools",
abstract = "Background:Evaluation of surgical skill competency is necessary as graduate medical education moves toward a competency-based curriculum. This study by the American Board of Orthopaedic Surgery (ABOS) and the Council of Orthopaedic Residency Directors (CORD) compares 2 web-based evaluation tools that assess the level of autonomy that is demonstrated by residents during surgical procedures in the operating room as measured by faculty.Methods:Two hundred and ninety-four residents from 16 orthopaedic surgery residency programs were evaluated by 370 faculty using 2 web-based evaluation tools in a crossover design in which residents requested faculty review of their surgical skills before starting a case. One thousand, one hundred and fifty Ottawa Surgical Competency Operating Room Evaluation (O-Score) assessments, which included a 9-question evaluation of 8 steps of the surgical procedure, were compared with 1,186 P-score evaluations, which included a single-question summative evaluation. Twenty-five different surgical procedures were evaluated.Results:There were no significant differences in rates of resident requests or faculty completion of the 2 scores. The most common surgical procedures that were assessed were total knee arthroplasty (n = 254, 11{\%}), carpal tunnel release (n = 191, 8{\%}), open reduction and internal fixation (ORIF) of stable hip fractures (n = 170, 7{\%}), ORIF of simple ankle fractures (n = 169, 7{\%}), and total hip arthroplasty (n = 166, 7{\%}). Both instruments disclosed significant differences in competency among entry, intermediate, and advanced-level residents. The findings support the construct validity of the evaluation method. The survey results indicated that >70{\%} of the faculty were confident that use of either the P-score or the O-score allowed them to distinguish a resident who can perform the surgery independently from one who needs additional training.Conclusions:This research has led to the modification of the O-score and the P-score into a combined OP-score instrument. The ABOS envisions that the OP-score instrument can be used with an expanded number of surgical procedures as a required element of residency training in the near future.Clinical Relevance:This study allows the profession of orthopaedic surgery education to take a leadership role in the measurement of competence for surgical skills for orthopaedic surgeons in residency training, an important clinically relevant topic to the practice of orthopaedic surgery.",
author = "{Van Heest}, {Ann E.} and Julie Agel and Ames, {S. Elizabeth} and Asghar, {Ferhan A.} and Harrast, {John J.} and Marsh, {J. Lawrence} and Patt, {Joshua C.} and Sterling, {Robert S.} and Peabody, {Terrance D}",
year = "2019",
month = "3",
day = "6",
doi = "10.2106/JBJS.17.01512",
language = "English (US)",
volume = "101",
journal = "The Journal of bone and joint surgery. American volume",
issn = "0021-9355",
publisher = "Journal of Bone and Joint Surgery Inc.",
number = "5",

}

Van Heest, AE, Agel, J, Ames, SE, Asghar, FA, Harrast, JJ, Marsh, JL, Patt, JC, Sterling, RS & Peabody, TD 2019, 'Resident Surgical Skills Web-Based Evaluation: A Comparison of 2 Assessment Tools', Journal of Bone and Joint Surgery - American Volume, vol. 101, no. 5, e18. https://doi.org/10.2106/JBJS.17.01512

Resident Surgical Skills Web-Based Evaluation : A Comparison of 2 Assessment Tools. / Van Heest, Ann E.; Agel, Julie; Ames, S. Elizabeth; Asghar, Ferhan A.; Harrast, John J.; Marsh, J. Lawrence; Patt, Joshua C.; Sterling, Robert S.; Peabody, Terrance D.

In: Journal of Bone and Joint Surgery - American Volume, Vol. 101, No. 5, e18, 06.03.2019.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Resident Surgical Skills Web-Based Evaluation

T2 - A Comparison of 2 Assessment Tools

AU - Van Heest, Ann E.

AU - Agel, Julie

AU - Ames, S. Elizabeth

AU - Asghar, Ferhan A.

AU - Harrast, John J.

AU - Marsh, J. Lawrence

AU - Patt, Joshua C.

AU - Sterling, Robert S.

AU - Peabody, Terrance D

PY - 2019/3/6

Y1 - 2019/3/6

N2 - Background:Evaluation of surgical skill competency is necessary as graduate medical education moves toward a competency-based curriculum. This study by the American Board of Orthopaedic Surgery (ABOS) and the Council of Orthopaedic Residency Directors (CORD) compares 2 web-based evaluation tools that assess the level of autonomy that is demonstrated by residents during surgical procedures in the operating room as measured by faculty.Methods:Two hundred and ninety-four residents from 16 orthopaedic surgery residency programs were evaluated by 370 faculty using 2 web-based evaluation tools in a crossover design in which residents requested faculty review of their surgical skills before starting a case. One thousand, one hundred and fifty Ottawa Surgical Competency Operating Room Evaluation (O-Score) assessments, which included a 9-question evaluation of 8 steps of the surgical procedure, were compared with 1,186 P-score evaluations, which included a single-question summative evaluation. Twenty-five different surgical procedures were evaluated.Results:There were no significant differences in rates of resident requests or faculty completion of the 2 scores. The most common surgical procedures that were assessed were total knee arthroplasty (n = 254, 11%), carpal tunnel release (n = 191, 8%), open reduction and internal fixation (ORIF) of stable hip fractures (n = 170, 7%), ORIF of simple ankle fractures (n = 169, 7%), and total hip arthroplasty (n = 166, 7%). Both instruments disclosed significant differences in competency among entry, intermediate, and advanced-level residents. The findings support the construct validity of the evaluation method. The survey results indicated that >70% of the faculty were confident that use of either the P-score or the O-score allowed them to distinguish a resident who can perform the surgery independently from one who needs additional training.Conclusions:This research has led to the modification of the O-score and the P-score into a combined OP-score instrument. The ABOS envisions that the OP-score instrument can be used with an expanded number of surgical procedures as a required element of residency training in the near future.Clinical Relevance:This study allows the profession of orthopaedic surgery education to take a leadership role in the measurement of competence for surgical skills for orthopaedic surgeons in residency training, an important clinically relevant topic to the practice of orthopaedic surgery.

AB - Background:Evaluation of surgical skill competency is necessary as graduate medical education moves toward a competency-based curriculum. This study by the American Board of Orthopaedic Surgery (ABOS) and the Council of Orthopaedic Residency Directors (CORD) compares 2 web-based evaluation tools that assess the level of autonomy that is demonstrated by residents during surgical procedures in the operating room as measured by faculty.Methods:Two hundred and ninety-four residents from 16 orthopaedic surgery residency programs were evaluated by 370 faculty using 2 web-based evaluation tools in a crossover design in which residents requested faculty review of their surgical skills before starting a case. One thousand, one hundred and fifty Ottawa Surgical Competency Operating Room Evaluation (O-Score) assessments, which included a 9-question evaluation of 8 steps of the surgical procedure, were compared with 1,186 P-score evaluations, which included a single-question summative evaluation. Twenty-five different surgical procedures were evaluated.Results:There were no significant differences in rates of resident requests or faculty completion of the 2 scores. The most common surgical procedures that were assessed were total knee arthroplasty (n = 254, 11%), carpal tunnel release (n = 191, 8%), open reduction and internal fixation (ORIF) of stable hip fractures (n = 170, 7%), ORIF of simple ankle fractures (n = 169, 7%), and total hip arthroplasty (n = 166, 7%). Both instruments disclosed significant differences in competency among entry, intermediate, and advanced-level residents. The findings support the construct validity of the evaluation method. The survey results indicated that >70% of the faculty were confident that use of either the P-score or the O-score allowed them to distinguish a resident who can perform the surgery independently from one who needs additional training.Conclusions:This research has led to the modification of the O-score and the P-score into a combined OP-score instrument. The ABOS envisions that the OP-score instrument can be used with an expanded number of surgical procedures as a required element of residency training in the near future.Clinical Relevance:This study allows the profession of orthopaedic surgery education to take a leadership role in the measurement of competence for surgical skills for orthopaedic surgeons in residency training, an important clinically relevant topic to the practice of orthopaedic surgery.

UR - http://www.scopus.com/inward/record.url?scp=85062639506&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85062639506&partnerID=8YFLogxK

U2 - 10.2106/JBJS.17.01512

DO - 10.2106/JBJS.17.01512

M3 - Article

C2 - 30845044

AN - SCOPUS:85062639506

VL - 101

JO - The Journal of bone and joint surgery. American volume

JF - The Journal of bone and joint surgery. American volume

SN - 0021-9355

IS - 5

M1 - e18

ER -