Classification of multiple annotator data using variational Gaussian process inference

In this paper we address supervised learning problems where, instead of having a single annotator who provides the ground truth, multiple annotators, usually with varying degrees of expertise, provide conflicting labels for the same sample. Once Gaussian Process classification has been adapted to this problem we propose and describe how Variational Bayes inference can be used to, given the observed labels, approximate the posterior distribution of the latent classifier and also estimate each annotator's reliability. In the experimental section, we evaluate the proposed method on both generated synthetic and real data, and compare it with state of the art crowdsourcing methods.