飞机异常检测
aircraft fuselage 飞机机身
Aircraft inspection and maintenance 飞机检查和维护
ABSTRACT
To ensure flight safety of aircraft structures, it is necessary to have regular maintenance using visual and nondestructive inspection (NDI) methods. In this paper, we propose an automatic image-based aircraft defect detection using Deep Neural Networks (DNNs). To the best of our knowledge, this is the first work for aircraft defect detection using DNNs. We perform a comprehensive evaluation of state of-the-art feature descriptors and show that the best performance is achieved by vgg-f DNN as feature extractor with a linear SVM classifier. To reduce the processing time, we propose to apply SURF key point detector to identify defect patch candidates. Our experiment results suggest that we can achieve over 96% accuracy at around 15s processing time for a high-resolution (20-megapixel) image on a laptop
为确保飞机结构的飞行安全，必须使用视觉和非破坏性检查（NDI）方法进行定期维护。 在本文中，我们提出了一种使用深度神经网络（DNN）进行基于图像的自动飞机缺陷检测。 据我们所知，这是使用DNN进行飞机缺陷检测的第一项工作。 我们对最先进的特征描述符进行了全面评估，并表明vgg-f DNN作为具有线性SVM分类器的特征提取器可以实现最佳性能。 为了减少处理时间，我们建议应用SURF关键点检测器来识别缺陷补丁候选者。 我们的实验结果表明，在笔记本电脑上拍摄高分辨率（20万像素）图像时，我们可以在大约15秒的处理时间内获得超过96％的准确度
1. INTRODUCTION
Aircraft inspection and maintenance is an essential to safe air transportation [1-3]. A fully automated system to monitor the structural health of an aircraft has the potential to reduce operating costs, increase flight safety and improve aircraft availability [4]. This paper makes a contribution to the field of automatic defect detection of an aircraft fuselage with computer vision techniques.
飞机检查和维护对安全航空运输至关重要[1-3]。 用于监控飞机结构健康状况的全自动系统有可能降低运营成本，提高飞行安全性并提高飞机可用性[4]。 本文利用计算机视觉技术为飞机机身的自动缺陷检测领域做出了贡献。
In [5], they research computer-simulated visual inspection (VI) and non-destructive inspection (NDI) tasks. However, these visual inspection tasks were performed by human inspectors who searched for defect manually. Our proposed algorithm is a completely automatic inspection.
在[5]中，他们研究了计算机模拟视觉检测（VI）和非破坏性检查（NDI）任务。 然而，这些视觉检查任务由人工检查员执行，他们手动搜索缺陷。 我们提出的算法是一种全自动检测。
[5] K. Latorella, A. Gramopadhye, P. Prabhu, C. Drury, M. Smith, and D. Shanahan, "Computer-simulated aircraft inspection tasks for off-line experimentation," in Proceedings of the Human Factors and Ergonomics Society Annual Meeting , 1992, pp. 92-96
In recent years, deep neural networks (DNN) have shown promising results in different classification tasks [6-10]. Although DNNs can be used to perform classification directly using the output of the last network layer, they can also be used as a feature extractor combined with a classifier [11].
近年来，深度神经网络（DNN）在不同的分类任务中显示出有希望的结果[6-10]。 虽然DNN可以用于直接使用最后一个网络层的输出进行分类，但它们也可以用作与分类器结合的特征提取器[11]。
In this paper, we investigate a classification system that employs a DNN, pretrained using natural images, to extract features suitable to another domain, i.e., aircraft fuselage defect detection, where there are few samples available. The contributions of this study are:
在本文中，我们研究了一种采用DNN的分类系统，该系统使用自然图像预训练，以提取适合于另一个域的特征，即飞机机身缺陷检测，其中可用的样本很少。 这项研究的贡献是：
1) To the best of our knowledge, this is the first work for automatic defect detection of aircraft fuselage using visual images and deep learning.
2) We propose a fast and accurate detection algorithm with selection of region of interest using SURF interest point extractor.
3) We propose techniques to handle washed and unwashed fuselage based on pre- and post-processing.
1）据我们所知，这是使用视觉图像和深度学习进行飞机机身自动缺陷检测的第一项工作。
2）我们提出了一种快速准确的检测算法，使用SURF兴趣点提取器选择感兴趣的区域。
surf
3）我们提出了基于预处理和后处理来处理洗涤和未洗涤机身的技术。
To the best of our knowledge, there is no previous work on automatic image-based aircraft defect detection. Image based defect detection has been investigated for other problems: In [12] X-ray images of metallic components are used as a non-destructive testing method, to detect the defects within casting components. In [13] they propose a deep convolutional neural network solution to the analysis of image data for the detection of rail surface defects. The images were obtained from many hours of automated video recordings. However, the image and defect characteristics of these problems are rather different from ours.
据我们所知，以前没有关于基于自动图像的飞机缺陷检测的工作。 已经针对其他问题研究了基于图像的缺陷检测：在[12]中，金属部件的X射线图像被用作非破坏性测试方法，以检测铸造部件内的缺陷。 在[13]中，他们提出了一种深度卷积神经网络解决方案，用于分析图像数据，以检测轨道表面缺陷。 这些图像是从许多小时的自动视频录制中获得的。 然而，这些问题的形象和缺陷特征与我们的不同。
[12] D. Mery and C. Arteta, "Automatic Defect Recognition in X-Ray Testing Using Computer Vision," in Applications of Computer Vision (WACV), 2017 IEEE Winter Conference on , 2017, pp. 1026-1035.
[13] S. Faghih-Roohi, S. Hajizadeh, A. Núñez, R. Babuska, and B. De Schutter, "Deep convolutional neural networks for detection of rail surface defects," in Neural Networks (IJCNN), 2016 International Joint Conference on , 2016, pp. 2584-2589.
The rest of the paper is organized as follows. In Section 2 we give a detailed explanation of our datasets. Section 3 explains the proposed algorithm and the DNN-derived features generated automatically from a dataset of fuselage images. The performance evaluation of the proposed algorithm is provided in Section 4, while Section 5 presents the conclusion and the feature work
本文的其余部分安排如下。 在第2节中，我们详细解释了我们的数据集。 第3节解释了所提出的算法以及从机身图像数据集自动生成的DNN衍生特征。 第4节提供了算法的性能评估，第5节给出了结论和特征工作
2. DATASETS
Our dataset images are taken in a straight view of the airplane fuselage. During the inspection, a drone can be used to capture these images automatically. Images are stored in JPEG format. All images have three color channels and 3888×5184 resolution. Some examples of aircraft fuselage images with defects are illustrated in Figure 1. For each image, a binary mask is created by an experienced inspector to represent defects. Considering
我们的数据集图像是在飞机机身的直视图中拍摄的。 在检查过程中，可以使用无人机自动捕获这些图像。 图像以JPEG格式存储。 所有图像都有三个颜色通道和3888×5184分辨率。 图1中示出了具有缺陷的飞机机身图像的一些示例。对于每个图像，由经验丰富的检查员创建二元掩模以表示缺陷。
3. METHODOLOGY
In this work, we propose a patch-based scheme for detection of defects. Specifically, we partition the image into 65x65 patches and classify each patch into defect / non-defect class. The classification is a two-step process: First, we compute a set of features for each patch; second, we build a classification model based on the extracted features.
在这项工作中，我们提出了一种基于补丁的方法来检测缺陷。 具体来说，我们将图像分区为65x65补丁，并将每个补丁分类为缺陷/非缺陷类。 分类分为两步：首先，我们为每个补丁计算一组特征; 第二，我们基于提取的特征构建分类模型。
For computing the discriminative features, we evaluate and compare a set of techniques including deep neural network, local descriptors, and texture features. Our experiments show that using the pretrained convolutional neural network (CNN) results in the best performance. For the classification step, we use SVM with a linear kernel.
为了计算判别特征，我们评估和比较一组技术，包括深度神经网络，局部描述符和纹理特征。 我们的实验表明，使用预训练卷积神经网络（CNN）可以获得最佳性能。 对于分类步骤，我们将SVM与线性内核一起使用。
In the experiment, we split the data into disjoint training and testing sets, in a manner that the data which is present in the training set is not allowed to be in the testing set. But in order to make these two sets completely disjoint, we employ 10-fold cross validation on the images rather than the patches, i.e. the patches of a particular image have the same crossvalidation index as their parent image. This approach prevents having highly correlated data in both training and testing sets, that results in high accuracy which is not the case
在实验中，我们将数据分成不相交的训练和测试集，其方式是不允许训练集中存在的数据在测试集中。 但是为了使这两组完全不相交，我们对图像而不是贴片采用10倍交叉验证，即特定图像的贴片具有与其父图像相同的交叉验证指数。 这种方法可以防止在训练集和测试集中都有高度相关的数据，从而导致高精度，而事实并非如此
Considering the high resolution of images in our dataset and the maximum allowed size of a patch, it needs high computational complexity to evaluate all the patches in a single image. Since this work addresses an industrial application, processing time is a critical factor. We propose to boost our algorithm by analyzing the region of interest which is extracted by SURF [15] detector.
考虑到我们的数据集中图像的高分辨率和补丁的最大允许大小，它需要高计算复杂度来评估单个图像中的所有补丁。 由于这项工作涉及工业应用，处理时间是一个关键因素。 我们建议通过分析由SURF [15]检测器提取的感兴趣区域来增强我们的算法。
Another critical problem in defect detection of airplane fuselage is about washed or unwashed fuselage. Unwashed fuselage with dirt causes some problems in detection of defects. We propose to extend our algorithm for both washed and unwashed conditions.
飞机机身缺陷检测中的另一个关键问题是洗涤或未洗涤的机身。 没有洗涤的机身带有污垢会导致检测缺陷的一些问题。 我们建议扩展我们的洗涤和未洗涤条件的算法。
3.1. Feature Extraction
As we will show in the results, using CNN as feature extractor achieves the best performance. Therefore, we have used a convolutional neural network (CNN) pre-trained on ImageNet as a feature extractor for our dataset. Transferring the knowledge of an existing CNN to a new domain has been studied and proved successful in several applications [11, 16, 17]. This approach is more appropriate for our application rather than fine-tuning the CNN [18] due to several reasons such as size and types of our dataset. Considering the limited size of our dataset, we propose to build a classifier model on top of the output (activations) of the hidden layers [19]. Furthermore, since the dataset (ImageNet) that was used to train CNN is quite different from our dataset, it is better to use the activations of the earlier layers of the network to construct the classifier. The block diagram of the proposed method for defect detection is shown in figure 3. As discussed in section 2, an equally balanced set of patches is used for training.
正如我们将在结果中显示的那样，使用CNN作为特征提取器可以实现最佳性能。因此，我们使用在ImageNet上预训练的卷积神经网络（CNN）作为我们数据集的特征提取器。将现有CNN的知识转移到新域已经在若干应用中被研究并证明是成功的[11,16,17]。由于我的数据集的大小和类型等多种原因，这种方法更适合我们的应用而不是微调CNN [18]。考虑到我们的数据集的大小有限，我们建议在隐藏层的输出（激活）之上构建分类器模型[19]。此外，由于用于训练CNN的数据集（ImageNet）与我们的数据集完全不同，因此最好使用网络早期层的激活来构建分类器。所提出的缺陷检测方法的框图如图3所示。如第2节所述，使用一组同等平衡的补丁进行训练。
In this work, we evaluate two CNN models related to ImageNet: AlexNet [6] and VGG-F networks [7]. As illustrated in figure 3, the nets comprise of eight layers; the first five are convolutional layers and the remaining three are fully connected layers. The size of the descriptors is 4096 for ‘fc6’ and ‘fc7’ layers and 1000 for ‘fc8’ layer. The input image of these models are images of 244 ×244 ×3 pixels. For this reason, our 65 × 65 pixel-patches were resized to the required size (all three channels are equal). Considering K neurons in fully connected layers, we consider the extracted layer as a feature vector
在这项工作中，我们评估了两个与ImageNet相关的CNN模型：AlexNet [6]和VGG-F网络[7]。 如图3所示，网由八层组成; 前五个是卷积层，其余三个是完全连接的层。 'fc6'和'fc7'层的描述符大小为4096，'fc8'层为1000。 这些模型的输入图像是244×244×3像素的图像。 出于这个原因，我们将65×65像素的贴片尺寸调整为所需的尺寸（所有三个通道都相同）。 考虑完全连接层中的K神经元，我们将提取的层视为特征向量
3.2. Boosting Defect Detection
As discussed in 3.1, the SVM classifier is fed with a set of discriminative features, extracted for each patch. Considering our high-resolution images (20-megapixel), there are lots of patches to be evaluated by the feature extractor and the classifier. To speed up the processing, one approach is to decrease the number of patches by increasing the patch size, but this affects the accuracy of detection. In general, increasing the patch size reduces the computational complexity but degrades the accuracy and vice versa. Therefore, there is a trade-off between the accuracy and the computational complexity to choose patch size. We have tested a variation of patch sizes from 20x20 to 100x100 pixels and the best results are achieved by patch size 65x65 pixels.
如3.1中所讨论的，SVM分类器被提供有一组判别特征，为每个补丁提取。 考虑到我们的高分辨率图像（20万像素），功能提取器和分类器需要评估许多补丁。 为了加快处理速度，一种方法是通过增加补丁大小来减少补丁数量，但这会影响检测的准确性。 通常，增加贴片尺寸会降低计算复杂度，但会降低精度，反之亦然。 因此，在准确度和计算复杂度之间存在选择补丁大小的权衡。 我们已经测试了从20x20到100x100像素的贴片尺寸的变化，并且通过贴片尺寸65x65像素实现了最佳结果。
Considering this patch size and the resolution of our images, it is a time-consuming task to evaluate all the patches within an image. We propose to boost our algorithm via enforcing the evaluation to some regions of interest. The regions of interest must include all the probable defect areas.
考虑到此修补程序大小和图像的分辨率，评估图像中的所有修补程序是一项耗时的任务。 我们建议通过对某些感兴趣的区域执行评估来提升我们的算法。 感兴趣的区域必须包括所有可能的缺陷区域。
Through our experiments, we found that, in most images, speeded up robust feature (SURF) is able to detect all the defect regions together with some normal regions which are similar to the defects. Therefore, we propose to apply SURF interest point detector to select some patches to be included in the evaluation procedure. A patch is included in the defect evaluation procedure if it contains at least one SURF key point. In this way, lots of homogenous regions of the fuselage are excluded from the evaluation step. Our results show that evaluating only the selected patches of the regions of interest can boost defect detection algorithm by a 6x speed-up. Figure 4 shows the block diagram of the boosted defect detection algorithm.
通过我们的实验，我们发现，在大多数图像中，加速鲁棒特征（SURF）能够检测所有缺陷区域以及一些与缺陷类似的正常区域。 因此，我们建议应用SURF兴趣点检测器来选择要包括在评估程序中的一些补丁。 如果补丁包含至少一个SURF关键点，则补丁包含在缺陷评估过程中。 以这种方式，机身的许多均匀区域被排除在评估步骤之外。 我们的结果表明，仅评估感兴趣区域的选定块可以通过6倍加速来增强缺陷检测算法。 图4显示了增强缺陷检测算法的框图。
3.3. Post-processing
As this work is an industrial application, we have to make the algorithm applicable for different conditions. Washing status of the aircraft is an important factor, which affects the defect detection procedure. As aircraft exterior cleaning procedure is time and effort consuming, it is usually done occasionally. As a result, an aircraft could be unwashed with dirty spots on it which mislead the defect detection process. In order to overcome this issue, we employ a user interface to choose between two different conditions of washed or unwashed aircraft. In the condition of washed aircraft, the detection pipeline is the same as discussed in section 3.2. But for an unwashed aircraft with dirty spots on the fuselage, we propose to apply a low-pass Gaussian filter to reduce the noise-like spots on the fuselage images, in a manner that has the minimum smoothing effect on the real defects. But, there is a trade-off between reducing the high-pass components of the image and retaining the defects thoroughly. Our approach to overcome this problem is to have a post-processing after classification of the patches, which is done by a similarity comparison of the adjacent patches. The intensity range of the patch is used as the similarity metric. Specifically, if a patch is detected as a defect patch, it is most likely to be in a defect region, so we also test its adjacent patches with a postprocessing scheme to ensure all the patches in a defect region are detected. If an adjacent patch satisfies the similarity threshold, it is classified into the defect class.
由于这项工作是工业应用，我们必须使算法适用于不同的条件。飞机的清洗状态是影响缺陷检测程序的重要因素。由于飞机外部清洁程序耗费时间和精力，通常偶尔进行。结果，飞机可能没有清洗，上面有脏点，误导了缺陷检测过程。为了克服这个问题，我们采用用户界面在两种不同的洗涤或未洗涤飞机条件之间进行选择。在水洗飞机的情况下，检测管道与3.2节中讨论的相同。但是对于机身上有污点的未经洗涤的飞机，我们建议采用低通高斯滤波器来减少机身图像上的类似噪声的斑点，其方式对真实缺陷具有最小的平滑效果。但是，在减少图像的高通分量和彻底保留缺陷之间需要进行权衡。我们克服该问题的方法是在对片进行分类之后进行后处理，这通过相邻片的相似性比较来完成。贴片的强度范围用作相似性度量。具体而言，如果将补丁检测为缺陷补丁，则最有可能位于缺陷区域，因此我们还使用后处理方案测试其相邻补丁，以确保检测到缺陷区域中的所有补丁。如果相邻的补丁满足相似性阈值，则将其分类为缺陷类。