Do baseline data influence standard setting for a clinical skills examination?

Diane B. Wayne*, Jeffrey H. Barsuk, Elaine Cohen, William C. McGaghie

*Corresponding author for this work

Research output: Contribution to journalArticle

27 Citations (Scopus)

Abstract

BACKGROUND: Clinical skills examinations (CSEs) are used frequently in medical education. Standard setting for CSEs may employ different methods with or without prior performance data. METHOD: An expert panel provided item-based (Angoff) and group-based (Hofstee) judgments about two central venous catheter insertion performance checklists on three occasions. Judges did not receive baseline performance data on the first occasion but did on occasions two and three. Judges' ratings were used to calculate a minimum passing standard (MPS) for the CSE. Interrater reliabilities and test-retest reliability (stability) were calculated. Passing standards are compared using performance data from a pilot study. RESULTS: Both methods produced reliable and stable data. Baseline data influenced the judges' decisions. Use of the Angoff method alone yielded lenient MPSs, whereas the Hofstee method alone yielded stringent MPSs. CONCLUSIONS: Standard setting is a critical component of CSEs. Baseline data influence judges' decisions. Averaging Angoff and Hofstee outcomes produced the optimal MPS.

Original languageEnglish (US)
JournalAcademic Medicine
Volume82
Issue number10 SUPPL.
DOIs
StatePublished - Jan 1 2007

Fingerprint

Clinical Competence
examination
performance
Central Venous Catheters
Medical Education
Checklist
Reproducibility of Results
rating
expert
education
Group

ASJC Scopus subject areas

  • Education

Cite this

@article{671f53d540e94f00844fac79ea59b990,
title = "Do baseline data influence standard setting for a clinical skills examination?",
abstract = "BACKGROUND: Clinical skills examinations (CSEs) are used frequently in medical education. Standard setting for CSEs may employ different methods with or without prior performance data. METHOD: An expert panel provided item-based (Angoff) and group-based (Hofstee) judgments about two central venous catheter insertion performance checklists on three occasions. Judges did not receive baseline performance data on the first occasion but did on occasions two and three. Judges' ratings were used to calculate a minimum passing standard (MPS) for the CSE. Interrater reliabilities and test-retest reliability (stability) were calculated. Passing standards are compared using performance data from a pilot study. RESULTS: Both methods produced reliable and stable data. Baseline data influenced the judges' decisions. Use of the Angoff method alone yielded lenient MPSs, whereas the Hofstee method alone yielded stringent MPSs. CONCLUSIONS: Standard setting is a critical component of CSEs. Baseline data influence judges' decisions. Averaging Angoff and Hofstee outcomes produced the optimal MPS.",
author = "Wayne, {Diane B.} and Barsuk, {Jeffrey H.} and Elaine Cohen and McGaghie, {William C.}",
year = "2007",
month = "1",
day = "1",
doi = "10.1097/ACM.0b013e318141f464",
language = "English (US)",
volume = "82",
journal = "Academic Medicine",
issn = "1040-2446",
publisher = "Lippincott Williams and Wilkins",
number = "10 SUPPL.",

}

Do baseline data influence standard setting for a clinical skills examination? / Wayne, Diane B.; Barsuk, Jeffrey H.; Cohen, Elaine; McGaghie, William C.

In: Academic Medicine, Vol. 82, No. 10 SUPPL., 01.01.2007.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Do baseline data influence standard setting for a clinical skills examination?

AU - Wayne, Diane B.

AU - Barsuk, Jeffrey H.

AU - Cohen, Elaine

AU - McGaghie, William C.

PY - 2007/1/1

Y1 - 2007/1/1

N2 - BACKGROUND: Clinical skills examinations (CSEs) are used frequently in medical education. Standard setting for CSEs may employ different methods with or without prior performance data. METHOD: An expert panel provided item-based (Angoff) and group-based (Hofstee) judgments about two central venous catheter insertion performance checklists on three occasions. Judges did not receive baseline performance data on the first occasion but did on occasions two and three. Judges' ratings were used to calculate a minimum passing standard (MPS) for the CSE. Interrater reliabilities and test-retest reliability (stability) were calculated. Passing standards are compared using performance data from a pilot study. RESULTS: Both methods produced reliable and stable data. Baseline data influenced the judges' decisions. Use of the Angoff method alone yielded lenient MPSs, whereas the Hofstee method alone yielded stringent MPSs. CONCLUSIONS: Standard setting is a critical component of CSEs. Baseline data influence judges' decisions. Averaging Angoff and Hofstee outcomes produced the optimal MPS.

AB - BACKGROUND: Clinical skills examinations (CSEs) are used frequently in medical education. Standard setting for CSEs may employ different methods with or without prior performance data. METHOD: An expert panel provided item-based (Angoff) and group-based (Hofstee) judgments about two central venous catheter insertion performance checklists on three occasions. Judges did not receive baseline performance data on the first occasion but did on occasions two and three. Judges' ratings were used to calculate a minimum passing standard (MPS) for the CSE. Interrater reliabilities and test-retest reliability (stability) were calculated. Passing standards are compared using performance data from a pilot study. RESULTS: Both methods produced reliable and stable data. Baseline data influenced the judges' decisions. Use of the Angoff method alone yielded lenient MPSs, whereas the Hofstee method alone yielded stringent MPSs. CONCLUSIONS: Standard setting is a critical component of CSEs. Baseline data influence judges' decisions. Averaging Angoff and Hofstee outcomes produced the optimal MPS.

UR - http://www.scopus.com/inward/record.url?scp=34748831085&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=34748831085&partnerID=8YFLogxK

U2 - 10.1097/ACM.0b013e318141f464

DO - 10.1097/ACM.0b013e318141f464

M3 - Article

C2 - 17895672

AN - SCOPUS:34748831085

VL - 82

JO - Academic Medicine

JF - Academic Medicine

SN - 1040-2446

IS - 10 SUPPL.

ER -