Listwise learning to rank by exploring unique ratings

Xiaofeng Zhu, Diego Klabjan

Research output: Chapter in Book/Report/Conference proceedingConference contribution

15 Scopus citations


In this paper, we propose new listwise learning-to-rank models that mitigate the shortcomings of existing ones. Existing listwise learning-to-rank models are generally derived from the classical Plackett-Luce model, which has three major limitations. (1) Its permutation probabilities overlook ties, i.e., a situation when more than one document has the same rating with respect to a query. This can lead to imprecise permutation probabilities and inefficient training because of selecting documents one by one. (2) It does not favor documents having high relevance. (3) It has a loose assumption that sampling documents at different steps is independent. To overcome the first two limitations, we model ranking as selecting documents from a candidate set based on unique rating levels in decreasing order. The number of steps in training is determined by the number of unique rating levels. More specifically, in each step, we apply multiple multi-class classification tasks to a document candidate set and choose all documents that have the highest rating from the document set. This is in contrast to taking one document step by step in the classical Plackett-Luce model. Afterward, we remove all of the selected documents from the document set and repeat until the remaining documents all have the lowest rating. We propose a new loss function and associated four models for the entire sequence of weighted classification tasks by assigning high weights to the selected documents with high ratings for optimizing Normalized Discounted Cumulative Gain (NDCG). To overcome the final limitation, we further propose a novel and efficient way of refining prediction scores by combining an adapted Vanilla Recurrent Neural Network (RNN) model with pooling given selected documents at previous steps. We encode all of the documents already selected by an RNN model. In a single step, we rank all of the documents with the same ratings using the last cell of the RNN multiple times. We have implemented our models using three settings: neural networks, neural networks with gradient boosting, and regression trees with gradient boosting. We have conducted experiments on four public datasets. The experiments demonstrate that the models notably outperform state-of-the-art learning-to-rank models.

Original languageEnglish (US)
Title of host publicationWSDM 2020 - Proceedings of the 13th International Conference on Web Search and Data Mining
PublisherAssociation for Computing Machinery, Inc
Number of pages9
ISBN (Electronic)9781450368223
StatePublished - Jan 20 2020
Event13th ACM International Conference on Web Search and Data Mining, WSDM 2020 - Houston, United States
Duration: Feb 3 2020Feb 7 2020

Publication series

NameWSDM 2020 - Proceedings of the 13th International Conference on Web Search and Data Mining


Conference13th ACM International Conference on Web Search and Data Mining, WSDM 2020
Country/TerritoryUnited States


  • Deep learning
  • Learning to rank
  • Listwise learning to rank
  • Recurrent neural network

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Software
  • Computer Science Applications


Dive into the research topics of 'Listwise learning to rank by exploring unique ratings'. Together they form a unique fingerprint.

Cite this