Development and verification of a taxonomy of assessment metrics for surgical technical skills

Connie C. Schmitz*, Debra DaRosa, Maura E. Sullivan, Shari Lynn Meyerson, Ken Yoshida, James R. Korndorffer

*Corresponding author for this work

Research output: Contribution to journalArticle

6 Citations (Scopus)

Abstract

PURPOSE: To create and empirically verify a taxonomy of metrics for assessing surgical technical skills, and to determine which types of metrics, skills, settings, learners, models, and instruments were most commonly reported in the technical skills assessment literature. METHOD: In 2011-2012, the authors used a rational analysis of existing and emerging metrics to create the taxonomy, and used PubMed to conduct a systematic literature review (2001-2011) to test the taxonomy's comprehensiveness and verifiability. Using 202 articles identified from the review, the authors classified metrics according to the taxonomy and coded data concerning their context and use. Frequencies (counts, percentages) were calculated for all variables. RESULTS: The taxonomy contained 12 objective and 4 subjective categories. Of 567 metrics identified in the literature, 520 (92%) were classified using the new taxonomy. Process metrics outnumbered outcome metrics by 8:1. The most frequent metrics were "time," "manual techniques" (objective and subjective), "errors," and "procedural steps." Only one new metric, "learning curve," emerged. Assessments of basic motor skills and skills germane to laparoscopic surgery dominated the literature. Novices, beginners, and intermediate learners were the most frequent subjects, and box trainers and virtual reality simulators were the most frequent models used for assessing performance. CONCLUSIONS: Metrics convey what is valued in human performance. This taxonomy provides a common nomenclature. It may help educators and researchers in procedurally oriented disciplines to use metrics more precisely and consistently. Future assessments should focus more on bedside tasks and open surgical procedures and should include more outcome metrics.

Original languageEnglish (US)
Pages (from-to)153-161
Number of pages9
JournalAcademic Medicine
Volume89
Issue number1
DOIs
StatePublished - Jan 1 2014

Fingerprint

taxonomy
Motor Skills
Learning Curve
virtual reality
PubMed
Terminology
Laparoscopy
surgery
performance
Research Personnel
educator
literature
learning

ASJC Scopus subject areas

  • Medicine(all)
  • Education

Cite this

Schmitz, C. C., DaRosa, D., Sullivan, M. E., Meyerson, S. L., Yoshida, K., & Korndorffer, J. R. (2014). Development and verification of a taxonomy of assessment metrics for surgical technical skills. Academic Medicine, 89(1), 153-161. https://doi.org/10.1097/ACM.0000000000000056
Schmitz, Connie C. ; DaRosa, Debra ; Sullivan, Maura E. ; Meyerson, Shari Lynn ; Yoshida, Ken ; Korndorffer, James R. / Development and verification of a taxonomy of assessment metrics for surgical technical skills. In: Academic Medicine. 2014 ; Vol. 89, No. 1. pp. 153-161.
@article{247d45b200a84698a666a9580f446fba,
title = "Development and verification of a taxonomy of assessment metrics for surgical technical skills",
abstract = "PURPOSE: To create and empirically verify a taxonomy of metrics for assessing surgical technical skills, and to determine which types of metrics, skills, settings, learners, models, and instruments were most commonly reported in the technical skills assessment literature. METHOD: In 2011-2012, the authors used a rational analysis of existing and emerging metrics to create the taxonomy, and used PubMed to conduct a systematic literature review (2001-2011) to test the taxonomy's comprehensiveness and verifiability. Using 202 articles identified from the review, the authors classified metrics according to the taxonomy and coded data concerning their context and use. Frequencies (counts, percentages) were calculated for all variables. RESULTS: The taxonomy contained 12 objective and 4 subjective categories. Of 567 metrics identified in the literature, 520 (92{\%}) were classified using the new taxonomy. Process metrics outnumbered outcome metrics by 8:1. The most frequent metrics were {"}time,{"} {"}manual techniques{"} (objective and subjective), {"}errors,{"} and {"}procedural steps.{"} Only one new metric, {"}learning curve,{"} emerged. Assessments of basic motor skills and skills germane to laparoscopic surgery dominated the literature. Novices, beginners, and intermediate learners were the most frequent subjects, and box trainers and virtual reality simulators were the most frequent models used for assessing performance. CONCLUSIONS: Metrics convey what is valued in human performance. This taxonomy provides a common nomenclature. It may help educators and researchers in procedurally oriented disciplines to use metrics more precisely and consistently. Future assessments should focus more on bedside tasks and open surgical procedures and should include more outcome metrics.",
author = "Schmitz, {Connie C.} and Debra DaRosa and Sullivan, {Maura E.} and Meyerson, {Shari Lynn} and Ken Yoshida and Korndorffer, {James R.}",
year = "2014",
month = "1",
day = "1",
doi = "10.1097/ACM.0000000000000056",
language = "English (US)",
volume = "89",
pages = "153--161",
journal = "Academic Medicine",
issn = "1040-2446",
publisher = "Lippincott Williams and Wilkins",
number = "1",

}

Schmitz, CC, DaRosa, D, Sullivan, ME, Meyerson, SL, Yoshida, K & Korndorffer, JR 2014, 'Development and verification of a taxonomy of assessment metrics for surgical technical skills', Academic Medicine, vol. 89, no. 1, pp. 153-161. https://doi.org/10.1097/ACM.0000000000000056

Development and verification of a taxonomy of assessment metrics for surgical technical skills. / Schmitz, Connie C.; DaRosa, Debra; Sullivan, Maura E.; Meyerson, Shari Lynn; Yoshida, Ken; Korndorffer, James R.

In: Academic Medicine, Vol. 89, No. 1, 01.01.2014, p. 153-161.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Development and verification of a taxonomy of assessment metrics for surgical technical skills

AU - Schmitz, Connie C.

AU - DaRosa, Debra

AU - Sullivan, Maura E.

AU - Meyerson, Shari Lynn

AU - Yoshida, Ken

AU - Korndorffer, James R.

PY - 2014/1/1

Y1 - 2014/1/1

N2 - PURPOSE: To create and empirically verify a taxonomy of metrics for assessing surgical technical skills, and to determine which types of metrics, skills, settings, learners, models, and instruments were most commonly reported in the technical skills assessment literature. METHOD: In 2011-2012, the authors used a rational analysis of existing and emerging metrics to create the taxonomy, and used PubMed to conduct a systematic literature review (2001-2011) to test the taxonomy's comprehensiveness and verifiability. Using 202 articles identified from the review, the authors classified metrics according to the taxonomy and coded data concerning their context and use. Frequencies (counts, percentages) were calculated for all variables. RESULTS: The taxonomy contained 12 objective and 4 subjective categories. Of 567 metrics identified in the literature, 520 (92%) were classified using the new taxonomy. Process metrics outnumbered outcome metrics by 8:1. The most frequent metrics were "time," "manual techniques" (objective and subjective), "errors," and "procedural steps." Only one new metric, "learning curve," emerged. Assessments of basic motor skills and skills germane to laparoscopic surgery dominated the literature. Novices, beginners, and intermediate learners were the most frequent subjects, and box trainers and virtual reality simulators were the most frequent models used for assessing performance. CONCLUSIONS: Metrics convey what is valued in human performance. This taxonomy provides a common nomenclature. It may help educators and researchers in procedurally oriented disciplines to use metrics more precisely and consistently. Future assessments should focus more on bedside tasks and open surgical procedures and should include more outcome metrics.

AB - PURPOSE: To create and empirically verify a taxonomy of metrics for assessing surgical technical skills, and to determine which types of metrics, skills, settings, learners, models, and instruments were most commonly reported in the technical skills assessment literature. METHOD: In 2011-2012, the authors used a rational analysis of existing and emerging metrics to create the taxonomy, and used PubMed to conduct a systematic literature review (2001-2011) to test the taxonomy's comprehensiveness and verifiability. Using 202 articles identified from the review, the authors classified metrics according to the taxonomy and coded data concerning their context and use. Frequencies (counts, percentages) were calculated for all variables. RESULTS: The taxonomy contained 12 objective and 4 subjective categories. Of 567 metrics identified in the literature, 520 (92%) were classified using the new taxonomy. Process metrics outnumbered outcome metrics by 8:1. The most frequent metrics were "time," "manual techniques" (objective and subjective), "errors," and "procedural steps." Only one new metric, "learning curve," emerged. Assessments of basic motor skills and skills germane to laparoscopic surgery dominated the literature. Novices, beginners, and intermediate learners were the most frequent subjects, and box trainers and virtual reality simulators were the most frequent models used for assessing performance. CONCLUSIONS: Metrics convey what is valued in human performance. This taxonomy provides a common nomenclature. It may help educators and researchers in procedurally oriented disciplines to use metrics more precisely and consistently. Future assessments should focus more on bedside tasks and open surgical procedures and should include more outcome metrics.

UR - http://www.scopus.com/inward/record.url?scp=84893637114&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84893637114&partnerID=8YFLogxK

U2 - 10.1097/ACM.0000000000000056

DO - 10.1097/ACM.0000000000000056

M3 - Article

VL - 89

SP - 153

EP - 161

JO - Academic Medicine

JF - Academic Medicine

SN - 1040-2446

IS - 1

ER -