Nonstationary AR modeling and constrained recursive estimation of the displacement field

In this paper, an approach for the constrained recursive estimation of the displacement vector field (DVF) in image sequences is presented. An estimate of the displacement vector at the working point is obtained by minimizing the linearized displaced frame difference based on a set of observations that belong to a causal neighborhood (mask). An expression for the variance of the linearization error (noise) is obtained. Because the estimation of the DVF is an ill-posed problem, the solution is constrained by considering an autoregressive (AR) model for the DVF. This AR model is first considered stationary, according to which the two components of the DVF are uncorrelated and each component is modeled by a 2-D discrete Markov sequence. A nonstationary AR model of the DVF is also considered by spatially adapting the model coefficients using a weighted LMS algorithm. Additional information about the solution is incorporated into the algorithm using a causal “oriented smoothness” constraint. Based on the above formulation, a set theoretic regularization approach is followed that results in a weighted constrained least-squares estimation of the DVF. The proposed algorithm shows an improved performance with respect to accuracy, robustness to occlusion, and smoothness of the estimated DVF when applied to typical videoconferencing scenes.