THE EFFECTS OF RATER SEVERITY AND RATER DISTRIBUTION ON EXAMINEES' ABILITY ESTIMATION FOR CONSTRUCTED-RESPONSE ITEMS

Zhen Wang, Lihua Yao

Research output: Contribution to journalArticlepeer-review

7 Scopus citations

Abstract

The current study used simulated data to investigate the properties of a newly proposed method (Yao's rater model) for modeling rater severity and its distribution under different conditions. Our study examined the effects of rater severity, distributions of rater severity, the difference between item response theory (IRT) models with rater effect and without rater effect, and the difference between the precision of the ability estimates for tests composed of only constructed-response (CR) items and for tests composed of multiple-choice (MC) and CR items combined. Our results indicate that rater severity and its distribution can increase the bias of examinees' ability estimates and lower test reliability. Moreover, using an IRT model with rater effects can substantially increase the precision in the examinees' ability estimates, especially when the test was composed of only CR items. We also compared Yao's rater model with Muraki's rater effect model (1993) in terms of ability estimation accuracy and rater parameter recovery. The estimation results from Yao's rater model using Markov chain Monte Carlo (MCMC) were better than those from Muraki's rater effect model using marginal maximum likelihood.

Original languageEnglish (US)
Pages (from-to)i-22
JournalETS Research Report Series
Volume2013
Issue number2
DOIs
StatePublished - Dec 2013

Keywords

  • IRT
  • IRT-based rater model
  • MCMC
  • distributions of rater severity

ASJC Scopus subject areas

  • Social Psychology
  • Education
  • Applied Psychology
  • Statistics, Probability and Uncertainty

Fingerprint

Dive into the research topics of 'THE EFFECTS OF RATER SEVERITY AND RATER DISTRIBUTION ON EXAMINEES' ABILITY ESTIMATION FOR CONSTRUCTED-RESPONSE ITEMS'. Together they form a unique fingerprint.

Cite this