工业缺陷检测数据扩充论文综述

持续更新中

前言

在工业视觉检测领域，由于深度学习，尤其是卷积神经网络的通用性以及性能都显著高于传统的数字图像处理方法，并且，由于由于其基于数据驱动，不需要设计规则，其适用性也高于传统方法。所以其成为目前工业视觉检测领域的研究热点。

但是深度学习是数据驱动的，因此，其需要大量的数据。而在工业产线上虽然可以取得大量的数据，但是大部分为重复的，价值密度低的无缺陷数据，而缺陷数据很少出现。因此深度学习的数据需求量和产线难出缺陷数据成为一组矛盾。

因此，很多研究者提出数据扩充方法，希望可以缓解或者解决数据量不足的问题。

论文归纳

1. Multistage GAN for Fabric Defect Detection

IEEE TIP Multistage GAN for Fabric Defect Detection

织物缺陷生成。两阶段生成，先生成缺陷区域，再生成背景区域，背景区域可以适用于不同时期的织物

Abstract: Fabric defect detection is an intriguing but challenging topic. Many methods have been proposed for fabric defect detection, but these methods are still suboptimal due to the complex diversity of both fabric textures and defects. In this paper, we propose a generative adversarial network (GAN)-based framework for fabric defect detection. Considering existing challenges in real-world applications, the proposed fabric defect detection system is capable of learning existing fabric defect samples and automatically adapting to different fabric textures during different application periods. Specifically, we customize a deep semantic segmentation network for fabric defect detection that can detect different defect types. Furthermore, we attempted to train a multistage GAN to synthesize reasonable defects in new defect-free samples. First, a texture-conditioned GAN is trained to explore the conditional distribution of defects given different texture backgrounds. Given a novel fabric, we aim to generate reasonable defective patches. Then, a GAN-based fusion network fuses the generated defects to specific locations. Finally, the well-trained multistage GAN continuously updates the existing fabric defect datasets and contributes to the fine-tuning of the semantic segmentation network to better detect defects under different conditions. Comprehensive experiments on various representative fabric samples are conducted to verify the detection performance of our proposed method.

2. Region- and Strength-Controllable GAN for Defect Generation and Segmentation in Industrial Images
   IEEE TII Region- and Strength-Controllable GAN for Defect Generation and Segmentation in Industrial Images

区域和强度控制缺陷生成，条件GAN，通过缺陷mask 控制缺陷区域，等同于图像修补。通过隐变量中的逻辑回归寻找缺陷到无缺陷的方向，并控制生成图像向无缺陷图像前进，以控制缺陷强度。

Abstract: Deep learning(DL for computer vision has achieved remarkable results based on massive, diverse, well-annotated training sets. However, it is difficult to collect defect datasets that cover all possible features, especially for small, weak defects. Therefore, a defect image generation method with controlling defect regions and strengths is proposed. Be regarded as image inpainting using generative adversarial network(GAN, the regions of generated defects are controlled using defect masks. Moreover, the defect direction vector is constructed in the latent variable space based on the feature continuity between defects and non-defects to control defect strength, enabling one-to-many correspondence between defect masks and images. Moreover, defect attention loss is also designed to force the generation model to focus on the defect regions. Experimentally, our method yields generated images of better quality and diversity and thus significantly improves defect segmentation performance (IOU: 63.20% and 61.86% on KSD and MHD dataset, especially for small, weak defects.

3. Semi-supervised defect classification of steel surface based on multi-training and generative adversarial network

Optics and Lasers in Engineering Semi-supervised defect classification of steel surface based on multi-training and generative adversarial network

通过GAN生成图像，并使用GAN的鉴别器和分类网络Resnet 判断并投票是否放置于训练数据集中。

Abstract: Defect inspection is very important for guaranteeing the surface quality of industrial steel products, but related methods are based primarily on supervised learning which requires ample labeled samples for training. However, there can be no doubt that inspecting defects on steel surface is always a data-limited task due to difficult sample collection and expensive expert labeling. Unlike the previous works in which only labeled samples are treated using supervised classifiers, we propose a semi-supervised learning (SSL) defect classification approach based on multi-training of two different networks: a categorized generative adversarial network (GAN) and a residual network. This method uses the GAN to generate a large number of unlabeled samples. And then the multi-training algorithm that uses two classifiers based on different learning strategies is proposed to integrate both labeled and unlabeled into SSL process. Finally, through the multiple training process, our SSL method can acquire higher accuracy and better robustness than the supervised one using only limited labeled samples. Experimental results clearly demonstrate that the effectiveness of our proposed method, achieving the classification accuracy of 99.56%.

4. Automated defect inspection system for metal surfaces based on deep learning and data augmentation
   Journal of Manufacturing Systems Automated defect inspection system for metal surfaces based on deep learning and data augmentation

使用条件VAE控制生成缺陷的类型

Abstract: Recent efforts to create a smart factory have inspired research that analyzes process data collected from Internet of Things (IOT) sensors, to predict product quality in real time. This requires an automatic defect inspection system that quantifies product quality data by detecting and classifying defects in real time. In this study, we propose a vision-based defect inspection system to inspect metal surface defects. In recent years, deep convolutional neural networks (DCNNs) have been used in many manufacturing industries and have demonstrated the excellent performance as a defect classification method. A sufficient amount of training data must be acquired, to ensure high performance using a DCNN. However, owing to the nature of the metal manufacturing industry, it is difficult to obtain enough data because some defects occur rarely. Owing to this imbalanced data problem, the generalization performance of the DCNN-based classification algorithm is lowered. In this study, we propose a new convolutional variational autoencoder (CVAE) and deep CNN-based defect classification algorithm to solve this problem. The CVAE-based data generation technology generates sufficient defect data to train the classification model. A conditional CVAE (CCVAE) is proposed to generate images for each defect type in a single CVAE model. We also propose a classifier based on a DCNN with high generalization performance using data generated from the CCVAE. In order to verify the performance of the proposed method, we performed experiments using defect images obtained from an actual metal production line. The results showed that the proposed method exhibited an excellent performance.

5. Defect-GAN: High-Fidelity Defect Synthesis for Automated Defect Inspection

WACV 2021 Defect-GAN: High-Fidelity Defect Synthesis for Automated Defect Inspection

条件生成，在无缺陷图像生成缺陷，Spatial & Categorical Control Map 控制缺陷类别和位置

Abstract: Automated defect inspection is critical for effective and efficient maintenance, repair, and operations in advanced manufacturing. On the other hand, automated defect inspection is often constrained by the lack of defect samples, especially when we adopt deep neural networks for this task. This paper presents Defect-GAN, an automated defect synthesis network that generates realistic and diverse defect samples for training accurate and robust defect inspection networks. Defect-GAN learns through defacement and restoration processes, where the defacement generates defects on normal surface images while the restoration removes defects to generate normal images. It employs a novel compositional layer-based architecture for generating realistic defects within various image backgrounds with different textures and appearances. It can also mimic the stochastic variations of defects and offer flexible control over the locations and categories of the generated defects within the image background. Extensive experiments show that Defect-GAN is capable of synthesizing various defects with superior diversity and fidelity. In addition, the synthesized defect samples demonstrate their effectiveness in training better defect inspection networks.

6. Mask2Defect: A Prior Knowledge-Based Data Augmentation Method for Metal Surface Defect Inspection

IEEE TII Mask2Defect: A Prior Knowledge Based Data Augmentation Method for Metal Surface Defect Inspection

条件生成CGAN，缺陷掩码控制缺陷特征，包括不同形状、严重程度、规模、旋转角度、空间位置和零件编号。两阶段，第一阶段掩码到渲染域缺陷，第二阶段渲染域到真实域

Abstract: In this paper, a new data augmentation algorithm, named Mask2Defect is proposed. Via prior knowledge based data infusing, this method is able to generate defects with varied features. Large volume of defects with different shapes, severities, scales, rotation angles, spatial locations, and part numbers can be generated in a controllable manner. These generated defects will work as teacher samples to fine-tune the inspection model, and automatically adapt it to a wider range of defects. To be specific, we first encode the prior knowledge into the teacher mask via the Industrial Prior Knowledge Encoder, and render the defect details according to the mask with the Mask-to-Defect Construction Network. Then, the Fake-to-Real Domain Transformation GAN is used to transform the rendered samples from the fake domain into the real defect domain. Experiments reveal that the synthesized image quality of our method outperforms the state-of-the-art generative methods, and the performance of the inspection model in defect classification and localization has also been improved by fine-tuned with the generated samples.

讨论

GoodFellow 在一次讨论中，说过基于GAN的数据扩充方法目前没有看到有效的结论。毕竟是大佬的意见，我开始也认为这个观点是对的。说GAN应用于数据扩充没有效果的原因可以理解为：本来数据信息那么多，你就算是用GAN生成大量的数据，也是现有信息在特征层的重组，并没有增加新的信息量，为什么可以提高检测模型的性能呢。

但是，我大量的实验表明，数据扩充对于缺陷检测是有一定程度的提升的。因此，我觉得对于基于GAN的工业视觉缺陷检测扩充有效的原因可能有：

1）GAN数据扩充是对现有特征的重组，相似的在像素层面上的CutMix，Mixup等方法类似，都是对现有的数据进行重组。这样的重组是否可以增加信息量呢，某种程度上可以说没有，但不能说他一定对检测效果没有提升，因为这么做可以增加采样的密度，提高分布的完整性，提升检测性能。

2）工业图像特征层次较低，比如缺陷的形状，位置，尺寸等特征中就蕴含了很多缺陷的信息，而这些特征可以被简单的设计与控制。也就是，对于缺陷的多样性，可以不用GAN来进行拟合，可以来自于场外因素，比如各种先验信息。而GAN只是起到各种信息综合并生成一张工业图像的生成器。

上述为我个人的观点，大家可以尽情喷并给出自己的理由。