Effect of Trainee Performance Data on Standard-Setting Judgments Using the Mastery Angoff Method

Stuart B. Prenner, William Craig McGaghie, Sarah Chuzi, Eric Cantey, Aashish K Didwania, Jeffrey H Barsuk

Research output: Contribution to journalArticle

1 Citation (Scopus)

Abstract

Background : Mastery learning in health professions education requires learners to learn and undergo assessment until they demonstrate a high level of competence. Setting defensible standards is key to accurately assessing educational outcomes in mastery learning. The Mastery Angoff method was proposed recently to set a minimum passing standard (MPS) for mastery learning curricula. However, it is unknown whether prior knowledge of trainee performance affects judges' decisions about setting an MPS using the Mastery Angoff method.

Objective : We sought to determine the effect of introducing baseline data about trainee performance on faculty judges' decisions about the Mastery Angoff MPS for a written examination.

Methods : We developed a mastery learning curriculum to train internal medicine residents and cardiology fellows about the correct interpretation of inpatient telemetry monitoring. All learners were required to meet or exceed an MPS on a 35-item written examination at the end of training. The MPS was set in 2017 by judges who used the item-based Mastery Angoff method without prior examinee performance information. The judges subsequently reevaluated the test items after receiving baseline data about trainee performance collected during pilot testing. Mastery Angoff MPSs with and without baseline performance data were compared.

Results : Twelve judges participated in the standard-setting exercise. The initial MPS was similar to the repeat MPS set after judges received trainee performance data (86.2% versus 86.9%, P = .23).

Conclusions : Prior knowledge about medical trainee performance data did not affect MPS as determined by the Mastery Angoff procedure.

Original languageEnglish (US)
Pages (from-to)301-305
Number of pages5
JournalJournal of graduate medical education
Volume10
Issue number3
DOIs
StatePublished - Jun 1 2018

Fingerprint

Learning
Curriculum
Telemetry
Health Occupations
Internal Medicine
Cardiology
Health Education
Mental Competency
Inpatients
Exercise

ASJC Scopus subject areas

  • Medicine(all)

Cite this

@article{8836bdbe4296406597f436c2e83d4a65,
title = "Effect of Trainee Performance Data on Standard-Setting Judgments Using the Mastery Angoff Method",
abstract = "Background : Mastery learning in health professions education requires learners to learn and undergo assessment until they demonstrate a high level of competence. Setting defensible standards is key to accurately assessing educational outcomes in mastery learning. The Mastery Angoff method was proposed recently to set a minimum passing standard (MPS) for mastery learning curricula. However, it is unknown whether prior knowledge of trainee performance affects judges' decisions about setting an MPS using the Mastery Angoff method.Objective : We sought to determine the effect of introducing baseline data about trainee performance on faculty judges' decisions about the Mastery Angoff MPS for a written examination.Methods : We developed a mastery learning curriculum to train internal medicine residents and cardiology fellows about the correct interpretation of inpatient telemetry monitoring. All learners were required to meet or exceed an MPS on a 35-item written examination at the end of training. The MPS was set in 2017 by judges who used the item-based Mastery Angoff method without prior examinee performance information. The judges subsequently reevaluated the test items after receiving baseline data about trainee performance collected during pilot testing. Mastery Angoff MPSs with and without baseline performance data were compared.Results : Twelve judges participated in the standard-setting exercise. The initial MPS was similar to the repeat MPS set after judges received trainee performance data (86.2{\%} versus 86.9{\%}, P = .23).Conclusions : Prior knowledge about medical trainee performance data did not affect MPS as determined by the Mastery Angoff procedure.",
author = "Prenner, {Stuart B.} and McGaghie, {William Craig} and Sarah Chuzi and Eric Cantey and Didwania, {Aashish K} and Barsuk, {Jeffrey H}",
year = "2018",
month = "6",
day = "1",
doi = "10.4300/JGME-D-17-00781.1",
language = "English (US)",
volume = "10",
pages = "301--305",
journal = "Journal of graduate medical education",
issn = "1949-8349",
publisher = "University of Finance and Management",
number = "3",

}

Effect of Trainee Performance Data on Standard-Setting Judgments Using the Mastery Angoff Method. / Prenner, Stuart B.; McGaghie, William Craig; Chuzi, Sarah; Cantey, Eric; Didwania, Aashish K; Barsuk, Jeffrey H.

In: Journal of graduate medical education, Vol. 10, No. 3, 01.06.2018, p. 301-305.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Effect of Trainee Performance Data on Standard-Setting Judgments Using the Mastery Angoff Method

AU - Prenner, Stuart B.

AU - McGaghie, William Craig

AU - Chuzi, Sarah

AU - Cantey, Eric

AU - Didwania, Aashish K

AU - Barsuk, Jeffrey H

PY - 2018/6/1

Y1 - 2018/6/1

N2 - Background : Mastery learning in health professions education requires learners to learn and undergo assessment until they demonstrate a high level of competence. Setting defensible standards is key to accurately assessing educational outcomes in mastery learning. The Mastery Angoff method was proposed recently to set a minimum passing standard (MPS) for mastery learning curricula. However, it is unknown whether prior knowledge of trainee performance affects judges' decisions about setting an MPS using the Mastery Angoff method.Objective : We sought to determine the effect of introducing baseline data about trainee performance on faculty judges' decisions about the Mastery Angoff MPS for a written examination.Methods : We developed a mastery learning curriculum to train internal medicine residents and cardiology fellows about the correct interpretation of inpatient telemetry monitoring. All learners were required to meet or exceed an MPS on a 35-item written examination at the end of training. The MPS was set in 2017 by judges who used the item-based Mastery Angoff method without prior examinee performance information. The judges subsequently reevaluated the test items after receiving baseline data about trainee performance collected during pilot testing. Mastery Angoff MPSs with and without baseline performance data were compared.Results : Twelve judges participated in the standard-setting exercise. The initial MPS was similar to the repeat MPS set after judges received trainee performance data (86.2% versus 86.9%, P = .23).Conclusions : Prior knowledge about medical trainee performance data did not affect MPS as determined by the Mastery Angoff procedure.

AB - Background : Mastery learning in health professions education requires learners to learn and undergo assessment until they demonstrate a high level of competence. Setting defensible standards is key to accurately assessing educational outcomes in mastery learning. The Mastery Angoff method was proposed recently to set a minimum passing standard (MPS) for mastery learning curricula. However, it is unknown whether prior knowledge of trainee performance affects judges' decisions about setting an MPS using the Mastery Angoff method.Objective : We sought to determine the effect of introducing baseline data about trainee performance on faculty judges' decisions about the Mastery Angoff MPS for a written examination.Methods : We developed a mastery learning curriculum to train internal medicine residents and cardiology fellows about the correct interpretation of inpatient telemetry monitoring. All learners were required to meet or exceed an MPS on a 35-item written examination at the end of training. The MPS was set in 2017 by judges who used the item-based Mastery Angoff method without prior examinee performance information. The judges subsequently reevaluated the test items after receiving baseline data about trainee performance collected during pilot testing. Mastery Angoff MPSs with and without baseline performance data were compared.Results : Twelve judges participated in the standard-setting exercise. The initial MPS was similar to the repeat MPS set after judges received trainee performance data (86.2% versus 86.9%, P = .23).Conclusions : Prior knowledge about medical trainee performance data did not affect MPS as determined by the Mastery Angoff procedure.

UR - http://www.scopus.com/inward/record.url?scp=85054990473&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85054990473&partnerID=8YFLogxK

U2 - 10.4300/JGME-D-17-00781.1

DO - 10.4300/JGME-D-17-00781.1

M3 - Article

VL - 10

SP - 301

EP - 305

JO - Journal of graduate medical education

JF - Journal of graduate medical education

SN - 1949-8349

IS - 3

ER -