Assessing Interrater Agreement via the Average Deviation Index Given a Variety of Theoretical and Methodological Problems

Kristin Smith-Crowe*, Michael J. Burke, Maryam Kouchaki, Sloane M. Signal

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

16 Scopus citations

Abstract

Currently, guidelines do not exist for applying interrater agreement indices to the vast majority of methodological and theoretical problems that organizational and applied psychology researchers encounter. For a variety of methodological problems, we present critical values for interpreting the practical significance of observed average deviation (AD) values relative to either single items or scales. For a variety of theoretical problems, we present null ranges for AD values, relative to either single items or scales, to be used for determining whether an observed distribution of responses within a group is consistent with a theoretically specified distribution of responses. Our discussion focuses on important ways to extend the usage of interrater agreement indices beyond problems relating to the aggregation of individual level data.

Original languageEnglish (US)
Pages (from-to)127-151
Number of pages25
JournalOrganizational Research Methods
Volume16
Issue number1
DOIs
StatePublished - Jan 1 2013

Keywords

  • aggregation
  • average deviation (AD)
  • interrater agreement
  • multilevel research
  • null distribution

ASJC Scopus subject areas

  • Decision Sciences(all)
  • Strategy and Management
  • Management of Technology and Innovation

Fingerprint Dive into the research topics of 'Assessing Interrater Agreement via the Average Deviation Index Given a Variety of Theoretical and Methodological Problems'. Together they form a unique fingerprint.

Cite this