Spatially-constrained similarity measure for large-scale object retrieval

Xiaohui Shen, Zhe Lin, Jonathan Brandt, Ying Wu

Research output: Contribution to journalArticlepeer-review

53 Scopus citations


One fundamental problem in object retrieval with the bag-of-words model is its lack of spatial information. Although various approaches are proposed to incorporate spatial constraints into the model, most of them are either too strict or too loose so that they are only effective in limited cases. In this paper, a new spatially-constrained similarity measure (SCSM) is proposed to handle object rotation, scaling, view point change and appearance deformation. The similarity measure can be efficiently calculated by a voting-based method using inverted files. During the retrieval process, object localization in the database images can also be simultaneously achieved using SCSM without post-processing. Furthermore, based on the retrieval and localization results of SCSM, we introduce a novel and robust re-ranking method with the κ-nearest neighbors of the query for automatically refining the initial search results. Extensive performance evaluations on six public data sets show that SCSM significantly outperforms other spatial models including RANSAC-based spatial verification, while κ-NN re-ranking outperforms most state-of-the-art approaches using query expansion. We also adapted SCSM for mobile product image search with an iterative algorithm to simultaneously extract the product instance from the mobile query image, identify the instance, and retrieve visually similar product images. Experiments on two product image search data sets show that our approach can robustly localize and extract the product in the query image, and hence drastically improve the retrieval accuracy over baseline methods.

Original languageEnglish (US)
Article number6678509
Pages (from-to)1229-1241
Number of pages13
JournalIEEE Transactions on Pattern Analysis and Machine Intelligence
Issue number6
StatePublished - Jun 2014


  • Object retrieval
  • bag-of-words
  • k-NN re-ranking
  • product image search
  • spatially-constrained similarity measure

ASJC Scopus subject areas

  • Software
  • Artificial Intelligence
  • Applied Mathematics
  • Computer Vision and Pattern Recognition
  • Computational Theory and Mathematics


Dive into the research topics of 'Spatially-constrained similarity measure for large-scale object retrieval'. Together they form a unique fingerprint.

Cite this