Attention-guided analysis of infrastructure damage with semi-supervised deep learning

Routine visual inspection is essential to maintain adequate safety and serviceability of civil infrastructures. Computer vision and machine learning based software techniques are becoming recognized methods that can potentially help the inspectors analyze the physical and functional condition of infrastructures from images and/or videos of the region of interest. More recently, deep learning approaches have been shown robust in identifying damages; yet these methods require precisely labeled large amount of training data for high accuracy complementary to visual assessment of inspectors. Especially in image segmentation operations, in which damages are subtracted from the image background for further analysis, there is a strong need to localize the damaged region prior to segmentation operation. However, available segmentation methods mostly focus on the latter step (i.e., delineation), and mis-localization of damaged regions causes accuracy drops. Inspired by the superiority of human cognitive system, where recognizing objects is simpler and more efficient than machine learning algorithms, which are superior to human in local tasks, this paper describes a novel method to dramatically improve the accuracy of the damage quantification (detection + segmentation) using an attention-guided technique. In the proposed method, a fast object detection model, Single Shot Detector (SSD) trained on VGG-16 base classifier architecture, performs a real-time crack and spall detection while working interactively with the human inspector to ensure recognition of the region of interest is well-performed. Upon the inspector's verification, happening in real-time, the detected damage region is used for damage segmentation for further analysis. This initial region of interest selection drastically lowers the computational cost, required amount of training data and reduces number of outliers. For optimal performance, a modified version of SegNet architecture was used for damage segmentation. Based on various performance criteria, the proposed attention-guided infrastructure damage analysis technique provides 30% more precision with a very minor sacrifice in computational speed compared to analysis without using attention guide.