Learning visual instance retrieval from failure: Efficient online local metric adaptation from negative samples

Jiahuan Zhou*, Ying Wu

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

1 Scopus citations


Existing visual instance retrieval (VIR) approaches attempt to learn a faithful global matching metric or discriminative feature embedding offline to cover enormous visual appearance variations, so as to directly use it online on various unseen probes for retrieval. However, their requirement for a huge set of positive training pairs is very demanding in practice and the performance is largely constrained for the unseen testing samples due to the severe data shifting issue. In contrast, this paper advocates a different paradigm: part of the learning can be performed online but with nominal costs, so as to achieve online metric adaptation for different query probes. By exploiting easily-available negative samples, we propose a novel solution to achieve the optimal local metric adaptation effectively and efficiently. The insight of our method is the local hard negative samples can actually provide tight constraints to fine tune the metric locally. Our local metric adaptation method is generally applicable to be used on top of any offline-learned baselines. In addition, this paper gives in-depth theoretical analyses of the proposed method to guarantee the reduction of the classification error both asymptotically and practically. Extensive experiments on various VIR tasks have confirmed our effectiveness and superiority.

Original languageEnglish (US)
Article number8721083
Pages (from-to)2858-2873
Number of pages16
JournalIEEE Transactions on Pattern Analysis and Machine Intelligence
Issue number11
StatePublished - Nov 1 2020


  • Visual instance retrieval
  • hard negative samples
  • online metric adaptation

ASJC Scopus subject areas

  • Software
  • Computer Vision and Pattern Recognition
  • Computational Theory and Mathematics
  • Artificial Intelligence
  • Applied Mathematics


Dive into the research topics of 'Learning visual instance retrieval from failure: Efficient online local metric adaptation from negative samples'. Together they form a unique fingerprint.

Cite this