We present a novel expert-model of Query-Driven Locally Adaptive (QDLA) Fisher faces for robust face recognition. For each query face, the proposed method first fits local Fisher models with different appearances. A hybrid expert model then integrates these local models and combines the classification results based on the estimated error rate for each local model. This approach addresses the large size recognition problem, where many local variations can not be adequately handled by a single global model in a single appearance space. To speed up the query process, Locality Sensitive Hash(LSH) is applied for fast nearest neighbor search. Experiments demonstrate the approach to be effective, robust, and fast for large size, multi-class, and multi-variance data sets.