In this paper, an adaptive multiple-input pel-recursive displacement estimation algorithm is presented. The displacement vector field (DVF) is estimated by minimizing the linearized displaced frame difference (DFD) using v submasks of causal mask around the working point. Then, v corresponding systems of equations are formed and the set theoretic regularization approach that is followed results in a weighted constrained least-squares estimation of the DVF by using information about the variance of the linearization error (noise) and the solution. The prior information about the solution is incorporated into the algorithm using a causal "oriented smoothness" constraint (OSC) which also provides a spatial prediction model for the estimated DVF. The improved performance of the proposed algorithm with respect to accuracy, robustness to occlusion and smoothness of the estimated DVF is demonstrated.