Since the appearance changes of the target jeopardize visual measurements and often lead to tracking failure in practice, trackers need to be adaptive to non-stationary appearances or to dynamically select features to track. However, this idea is threatened by the risk of adaptation drift that roots in its ill-posed nature, unless good constraints are imposed. Different from most existing adaptation schemes, we enforce three novel constraints for the optimal adaptation: (1) negative data, (2) bottom-up pair-wise data constraints, and (3) adaptation dynamics. Substantializing the general adaptation problem as a subspace adaptation problem, this paper gives a closed-form solution as well as a practical iterative algorithm. Extensive experiments have shown that the proposed approach can largely alleviate adaptation drift and achieve better tracking results.