Neurology objective structured clinical examination reliability using generalizability theory

Angela D. Blood*, Yoon Soo Park, Rimas V. Lukas, James R. Brorson

*Corresponding author for this work

Research output: Contribution to journalArticle

8 Scopus citations

Abstract

Objectives: This study examines factors affecting reliability, or consistency of assessment scores, from an objective structured clinical examination (OSCE) in neurology through generalizability theory (G theory). Methods: Data include assessments from a multistation OSCE taken by 194 medical students at the completion of a neurology clerkship. Facets evaluated in this study include cases, domains, and items. Domains refer to areas of skill (or constructs) that the OSCE measures. G theory is used to estimate variance components associated with each facet, derive reliability, and project the number of cases required to obtain a reliable (consistent, precise) score. Results: Reliability using G theory is moderate (Φ coefficient 0.61, G coefficient 0.64). Performance is similar across cases but differs by the particular domain, such that the majority of variance is attributed to the domain. Projections in reliability estimates reveal that students need to participate in 3 OSCE cases in order to increase reliability beyond the 0.70 threshold. Conclusions: This novel use of G theory in evaluating an OSCE in neurology provides meaningful measurement characteristics of the assessment. Differing from prior work in other medical specialties, the cases students were randomly assigned did not influence their OSCE score; rather, scores varied in expected fashion by domain assessed.

Original languageEnglish
Pages (from-to)1623-1629
Number of pages7
JournalNeurology
Volume85
Issue number18
DOIs
StatePublished - Nov 3 2015

ASJC Scopus subject areas

  • Clinical Neurology

Fingerprint Dive into the research topics of 'Neurology objective structured clinical examination reliability using generalizability theory'. Together they form a unique fingerprint.

  • Cite this