Sensible scenes: visual understanding of complex structures through causal analysis

An important result of visual understanding is an explanation of a scene's causal structure: How action - usually motion - is originated, constrained, and prevented, and how this determines what will happen in the immediate future. To be useful for a purposeful agent, these explanations must also capture the scene in terms of the functional properties of its objects - their purposes, uses, and affordances for manipulation. Design knowledge describes how the world is organized to suit these functions, and causal knowledge describes how these arrangements work. We have been exploring the hypothesis that vision is an explanatory process in which causal and functional reasoning plays an intimate role in mediating the activity of low-level visual processes. In particular, we have explored two of the consequences of this view for the construction of purposeful vision systems: Causal and design knowledge can be used to 1) drive focus of attention, and 2) choose between ambiguous image interpretations. Both principles are at work in SPROCKET, a system which visually explores simple machines, integrating diverse visual clues into an explanation of a machine's design and function.