RI: Small:Modeling and Learning Visual Similarities under Adverse Visual Conditions

Project: Research project

Project Details


RI: Small: Modeling and Learning Visual Similarities in the Wild
Di�erent from visual classi�cation, visual regression pursues the mapping from the high-
dimensional visual space to another continuous space for more �ne-grained visual inference. It
has wide applications in vision tasks, but it remains to be a very challenging problem. Traditional
approaches attempt to use a holistic parameterized function to �t the training data directly, and
thus they are painstaking o�-line learning processes. With the rapid growth of computational
power, it is very worthwhile to explore a new methodology that shifts part of the learning from
o�-line to on-line computation. The core idea is to steer the global regression on-the-
y to local
regression that is only performed on a small set of exemplars retrieved from a large exemplar
database. Studying this "regression-by-search" approach is the main focus of this proposal.
Despite the its
exibility in learning and adaptation, the study of this new approach is still
in its early stage, and many fundamental issues need to be addressed, including visual similarity,
regression modeling, and handling massive visual data, etc. The long-term goal of this project
is to lay the foundation of modeling and learning space metric for visual regression, by pursuing
innovative "regression-by-search" approaches to providing
exible and adaptive solutions to
di�cult visual regression tasks, to facilitating analyzable and explainable regression, and to
empowering many emerging and versatile applications. The proposed research will lead to more
e�ective methods for visual regression.
Key Words: computer vision; visual regression; image super resolution; gaze estimation
Intellectual Merit: The major objective is to integrate modeling and learning for visual
regression, and to provide methods and tools for real-life applications. It is focused on:
� Modeling the integration of visual regression and search for analyzable and explainable
regression. This novel modeling decomposes the complex regression into several analyzable
components including metric learning for space alignment, visual similarity, robust visual
search, local reconstruction and transferring.
� Learning space metric for space alignment for visual similarity and visual regression. It
pursues to answer an important question: under what condition and how the local recon-
struction can be transferred? The novel research is to explore and model the structural
relationship between the input and output spaces, which we call space alignment. It facili-
tates to improve the quality of visual search, and to better regularize local reconstruction.
� Two solid case studies on challenging tasks for validation. One is single-image super
resolution, which is a very di�cult ill-posed visual regression problem. And the other is
remote accurate gaze estimation. The proposed research is validated and evaluated on
these two case studies.
Broader Impact: The project is expected to signi�cantly advance the research of visual regres-
sion and visual similarity, and to result in an important enabling technology for a wide range
of applications including visual search, intelligent surveillance and security, human-computer
interaction, visual communications, etc. In addition, the research of image SR and gaze is also
helpful for the visually impaired for better visual sensing abilities.
This research program will contribute to education through curriculum development, student
involvements, and workshops and tutorials outside the vision
Effective start/end date9/1/168/31/21


  • National Science Foundation (IIS-1619078)


Explore the research topics touched on by this project. These labels are generated based on the underlying awards/grants. Together they form a unique fingerprint.