Behavioral researches have shown that the visual function can be partly restored by phosphene-based prosthetic vision for the non-congenital blinds. However, the early visual processing mechanisms of phosphene object recognition is still unclear. This paper aimed to investigate the electro-neurophysiology underlying the phosphene face and non-face recognition. The modulations of latency and amplitude of N170 component in the event-related potential (ERP) were analyzed. Our preliminary results showed that (1) both normal and phosphene face stimuli could elicit prominent N170; nevertheless, phosphene stimuli caused notable latency delay and amplitude suppression on N170 compared with normal stimuli and (2) under phosphene non-face stimuli, a slight but significant latency delay occurred compared with normal stimuli, while amplitude suppression was not observed. Therefore, it was suggested that (1) phosphene perception caused a disruption of the early visual processing for non-canonical images of objects, which was more profound in phosphene face processing; (2) the face-specific processing was reserved under prosthetic vision and (3) holistic processing was the major stage in early visual processing of phosphene face recognition, while part-based processing was attenuated due to the loss of the details.