On the Analysis on Multi-Rater Ordinal Data
May 9 1993
A framework is proposed for the analysis of ordinal categorical data when ratings from several judges are available. Emphasis focuses on the tasks of estimating quantiles of individual items, regressing item quantiles on observed covariates, and comparing the relative efficiency of individual raters. To illustrate the sue of the model framework in accomplishing these tasks, two examples are provided. In the first, a statistical essay grader is derived from variables extracted from a grammar checker. The grader's performance is then evaluated relative to expert human graders. In the second example, standard receiver operator characteristic (ROC) analysis is extended to incorporate interjudge variability.
Keywords:Categorical data, ROC analysis, hierarchical models, Bayesian inference, Gibbs sampling, Kappa statistic, log-linear models