Decoding what people see from where they look: Predicting visual stimuli from scanpaths

Moran Cerf*, Jonathan Harel, Alex Huth, Wolfgang Einhäuser, Christof Koch

*Corresponding author for this work

Research output: Contribution to journalConference articlepeer-review

10 Scopus citations


Saliency algorithms are applied to correlate with the overt attentional shifts, corresponding to eye movements, made by observers viewing an image. In this study, we investigated if saliency maps could be used to predict which image observers were viewing given only scanpath data. The results were strong: in an experiment with 441 trials, each consisting of 2 images with scanpath data - pooled over 9 subjects - belonging to one unknown image in the set, in 304 trials (69%) the correct image was selected, a fraction significantly above chance, but much lower than the correctness rate achieved using scanpaths from individual subjects, which was 82.4%. This leads us to propose a new metric for quantifying the importance of saliency map features, based on discriminability between images, as well as a new method for comparing present saliency map efficacy metrics. This has potential application for other kinds of predictions, e.g., categories of image content, or even subject class.

Original languageEnglish (US)
Pages (from-to)15-26
Number of pages12
JournalLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume5395 LNAI
StatePublished - 2009
Event5th International Workshop on Attention in Cognitive Systems, WAPCV 2008 - Fira, Santorini, Greece
Duration: May 12 2008May 12 2008

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)


Dive into the research topics of 'Decoding what people see from where they look: Predicting visual stimuli from scanpaths'. Together they form a unique fingerprint.

Cite this