Modeling Ordinal Scale Disagreement

In many instances, it may be of interest to examine the patterns of disagreement between two raters. In this work, an approach for modeling chance-adjusted disagreement is proposed. It is shown, by examining a hierarchy of log-linear models, how one can model ordinal scale disagreement in a manner analogous to the analysis of association in a contingency table. A series of specific hypotheses based on the magnitude and direction of the discrepancies of the ratings is considered. An example based on data from the National Institute of Mental Health (NIMH) Clinical Research Branch Collaborative Program on the Psychobiology of Depression is presented to illustrate the approach.