ImageInpaint:图像修复

文章目录

  • 前言
  • 传统方法
  • 深度学习
  • 论文研读
    • 方法
      • 网络结构
  • 参考

前言

传统方法

Year Proceeding Title Tag
2000 SIGGRAPH 2000 Image Inpainting [pdf] Diffusion-based
2001 TIP 2001 Filling-in by joint interpolation of vector fields and gray levels [pdf] Diffusion-based
2001 CVPR 2001 Navier-stokes, fluid dynamics, and image and video inpainting [pdf]
2001 SIGGRAPH 2001 Image Quilting for Texture Synthesis and Transfer [pdf]
2001 SIGGRAPH 2001 Synthesizing Natural Textures [pdf]
2002 EJAM 2002 Digital inpainting based on the mumford–shah–euler image model [pdf] Diffusion-based
2003 CVPR 2003 Object removal by exemplar-based inpainting [pdf]
2003 TIP 2003 Simultaneous structure and texture image inpainting [pdf] Diffusion-based
2003 TIP 2003 Structure and Texture Filling-In of Missing Image Blocks in Wireless Transmission and Compression Applications [pdf]
2003 ICCV 2003 Learning How to Inpaint from Global Image Statistics [pdf] Diffusion-based
2003 TOG 2003 Fragment-based image completion [pdf] Patch-based
2004 TIP 2004 Region Filling and Object Removal by Exemplar-Based Image Inpainting [pdf] Patch-based; Inpainting order
2004 TPAMI 2004 Space-Time Video Completion [pdf]
2005 SIGGRAPH 2005 Image Completion with Structure Propagation [pdf] Patch-based
2006 ISCS 2006 Image Compression with Structure Aware Inpainting [pdf]
2007 TOG 2007 Scene completion using millions of photographs [pdf]
2007 CSVT 2007 Image Compression With Edge-Based Inpainting [pdf] Diffusion-based
2008 CVPR 2008 Summarizing Visual Data Using Bidirectional Similarity [pdf]
2009 SIGGRAPH 2009 PatchMatch: a randomized correspondence algorithm for structural image editing [pdf] Patch-based
2010 TIP 2010 Image inpainting by patch propagation using patch sparsity [pdf] Patch-based
2011 FTCGV 2011 Structured learning and prediction in computer vision [pdf]
2011 ICIP 2011 Examplar-based inpainting based on local geometry [pdf] Inpainting order
2012 TOG 2012 Combining inconsistent images using patch-based synthesis[pdf] Patch-based
2014 TOG 2014 Image completion using Planar structure guidance [pdf] Patch-based
2014 TVCG 2014 High-Quality Real-Time Video Inpainting with PixMix [pdf] Video
2014 SIAM 2014 Video inpainting of complex scenes [pdf] Video
2015 TIP 2015 Annihilating Filter-Based Low-Rank Hankel Matrix Approach for Image Inpainting [pdf]
2015 TIP 2015 Exemplar-Based Inpainting: Technical Review and New Heuristics for Better Geometric Reconstructions [pdf]
2016 TOG 2016 Temporally coherent completion of dynamic video [pdf] Video

深度学习

Year Proceeding Title Tag
2012 NIPS 2012 Image denoising and inpainting with deep neural networks [pdf]
2014 GCPR 2014 Mask-specific inpainting with deep neural networks [pdf]
2014 NIPS 2014 Deep Convolutional Neural Network for Image Deconvolution [pdf]
2015 NIPS 2015 Shepard Convolutional Neural Networks [pdf] [code]
2016 CVPR 2016 Context Encoders: Feature Learning by Inpainting [pdf] [code]
2016 SIGGRAPH 2016 High-resolution multi-scale neural texture synthesis [pdf]
2017 CVPR 2017 Semantic image inpainting with deep generative models [pdf] [code]
2017 CVPR 2017 High-resolution image inpainting using multi-scale neural patch synthesis [pdf] [code]
2017 CVPR 2017 Generative Face Completion [pdf] [code]
2017 SIGGRAPH 2017 Globally and Locally Consistent Image Completion [pdf] [code]
2018 CVPR 2018 Generative Image Inpainting with Contextual Attention [pdf] [code]
2018 CVPR 2018 Natural and Effective Obfuscation by Head Inpainting [pdf]
2018 CVPR 2018 Eye In-Painting With Exemplar Generative Adversarial Networks [pdf] [code]
2018 CVPR 2018 UV-GAN: Adversarial Facial UV Map Completion for Pose-invariant Face Recognition [pdf]
2018 CVPR 2018 Disentangling Structure and Aesthetics for Style-aware Image Completion [pdf]
2018 ECCV 2018 Image Inpainting for Irregular Holes Using Partial Convolutions [pdf] [code]
2018 ECCV 2018 Contextual-based Image Inpainting: Infer, Match, and Translate [pdf]
2018 ECCV 2018 Shift-Net: Image Inpainting via Deep Feature Rearrangement [pdf] [code]
2018 NIPS 2018 Image Inpainting via Generative Multi-column Convolutional Neural Networks [pdf] [code]
2018 TOG 2018 Faceshop: Deep sketch-based face image editing [pdf]
2018 ACM MM 2018 Structural inpainting [pdf]
2018 ACM MM 2018 Semantic Image Inpainting with Progressive Generative Networks [pdf] [code]
2018 BMVC 2018 SPG-Net: Segmentation Prediction and Guidance Network for Image Inpainting [pdf]
2018 BMVC 2018 MC-GAN: Multi-conditional Generative Adversarial Network for Image Synthesi [pdf] [code]
2018 ACCV 2018 Face Completion iwht Semantic Knowledge and Collaborative Adversarial Learning [pdf]
2018 ICASSP 2018 Edge-Aware Context Encoder for Image Inpainting [pdf]
2018 ICPR 2018 Deep Structured Energy-Based Image Inpainting [pdf] [code]
2018 AISTATS 2019 Probabilistic Semantic Inpainting with Pixel Constrained CNNs [pdf]
2019 ICRA 2019 Empty Cities: Image Inpainting for a Dynamic-Object-Invariant Space [pdf]
2019 AAAI 2019 Video Inpainting by Jointly Learning Temporal Structure and Spatial Details [pdf] Video
2019 CVPR 2019 Pluralistic Image Completion [pdf] [code] [project]
2019 CVPR 2019 Deep Reinforcement Learning of Volume-guided Progressive View Inpainting for 3D Point Scene Completion from a Single Depth Image [pdf]
2019 CVPR 2019 Foreground-aware Image Inpainting [pdf]
2019 CVPR 2019 Privacy Protection in Street-View Panoramas using Depth and Multi-View Imagery [pdf]
2019 CVPR 2019 Learning Pyramid-Context Encoder Network for High-Quality Image Inpainting [pdf] [code]
2019 CVPR 2019 Deep Flow-Guided Video Inpainting [pdf] [project] Video
2019 CVPR 2019 Deep video inapinting [pdf] Video
2019 CVPRW 2019 VORNet: Spatio-temporally Consistent Video Inpainting for Object Removal [pdf] Video
2019 TNNLS 2019 PEPSI++: Fast and Lightweight Network for Image Inpainting [pdf]
2019 IJCAI 2019 MUSICAL: Multi-Scale Image Contextual Attention Learning for Inpainting [pdf]
2019 IJCAI 2019 Generative Image Inpainting with Submanifold Alignment [pdf]
2019 ACM MM 2019 Progressive Image Inpainting with Full-Resolution Residual Network [pdf] [code]
2019 ACM MM 2019 Deep Fusion Network for Image Completion [pdf] [code]
2019 ACM MM 2019 GAIN: Gradient Augmented Inpainting Network for Irregular Holes [pdf]
2019 ACM MM 2019 Single-shot Semantic Image Inpainting with Densely Connected Generative Networks [pdf]
2019 ICCVW 2019 EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning [pdf] [code]
2019 ICCV 2019 Coherent Semantic Attention for Image Inpainting [pdf] [code]
2019 ICCV 2019 StructureFlow: Image Inpainting via Structure-aware Appearance Flow [pdf] [code]
2019 ICCV 2019 Progressive Reconstruction of Visual Structure for Image Inpainting [pdf] [code]
2019 ICCV 2019 Localization of Deep Inpainting Using High-Pass Fully Convolutional Network [pdf]
2019 ICCV 2019 Image Inpainting with Learnable Bidirectional Attention Maps [pdf] [code]
2019 ICCV 2019 Free-Form Image Inpainting with Gated Convolution [pdf] [project]
2019 ICCV 2019 FiNet: Compatible and Diverse Fashion Image Inpainting [pdf] Fashion
2019 ICCV 2019 SC-FEGAN: Face Editing Generative Adversarial Network with User’s Sketch and Color [pdf] [code] Face
2019 ICCV 2019 Human Motion Prediction via Spatio-Temporal Inpainting [pdf] [code] Motion
2019 ICCV 2019 Copy-and-Paste Networks for Deep Video Inpainting [pdf] [code] Video
2019 ICCV 2019 Onion-Peel Networks for Deep Video Completion [pdf] [code] Video
2019 ICCV 2019 Free-form Video Inpainting with 3D Gated Convolution and Temporal PatchGAN [pdf] [code] Video
2019 ICCV 2019 An Internal Learning Approach to Video Inpainting [pdf] Video
2019 ICCV 2019 Vision-Infused Deep Audio Inpainting [pdf] [code] Audio
2019 AAAI 2020 Region Normalization for Image Inpainting [pdf] [code]
2019 AAAI 2020 Learning to Incorporate Structure Knowledge for Image Inpainting [pdf] [code]
2020 CVPR 2020 Prior Guided GAN Based Semantic Inpainting [pdf]
2020 CVPR 2020 UCTGAN: Diverse Image Inpainting based on Unsupervised Cross-Space Translation [pdf]
2020 CVPR 2020 Recurrent Feature Reasoning for Image Inpainting [pdf] [code]
2020 CVPR 2020 Contextual Residual Aggregation for Ultra High-Resolution Image Inpainting [pdf] [code]
2020 CVPR 2020 3D Photography using Context-aware Layered Depth Inpainting [pdf] [code]
2020 CVPR 2020 Learning Oracle Attention for High-fidelity Face Completion [pdf]
2020 ECCV 2020 Rethinking Image Inpainting via a Mutual Encoder-Decoder with Feature Equalizations [pdf] [code]
2020 ECCV 2020 Short-Term and Long-Term Context Aggregation Network for Video Inpainting Video
2020 ECCV 2020 Learning Object Placement by Inpainting for Compositional Data Augmentation
2020 ECCV 2020 Learning Joint Spatial-Temporal Transformations for Video Inpainting [pdf] [code] Video
2020 ECCV 2020 High-Resolution Image Inpainting with Iterative Confidence Feedback and Guided Upsampling [pdf]
2020 ECCV 2020 DVI: Depth Guided Video Inpainting for Autonomous Driving [pdf] [code] Video
2020 ECCV 2020 VCNet: A Robust Approach to Blind Image Inpainting [pdf]
2020 ECCV 2020 Guidance and Evaluation: Semantic-Aware Image Inpainting for Mixed Scenes [pdf]
2021 WACV 2021 Hyperrealistic Image Inpainting with Hypergraphs [pdf] [code]
2021 CVPR 2021 Generating Diverse Structure for Image Inpainting With Hierarchical VQ-VAE [pdf] [code]
2021 CVPR 2021 Image Inpainting with External-internal Learning and Monochromic Bottleneck [pdf] [code]
2021 CVPR 2021 PD-GAN: Probabilistic Diverse GAN for Image Inpainting [pdf] [code]
2021 CVPR 2021 Image Inpainting Guided by Coherence Priors of Semantics and Textures [pdf]
2021 CVPR 2021 FaceInpainter: High Fidelity Face Adaptation to Heterogeneous Domains [pdf]
2021 CVPR 2021 TransFill: Reference-guided Image Inpainting by Merging Multiple Color and Spatial Transformations [pdf] [code]
2021 CVPR 2021 Prior Based Human Completion [pdf] Human completion
2021 IJCAI 2021 Context-Aware Image Inpainting with Learned Semantic Priors [pdf] [code]
2021 IJCAI 2021 Noise Doesn’t Lie: Towards Universal Detection of Deep Inpainting [pdf] Inpainting detection
2021 TCSVT 2021 IID-Net: Image Inpainting Detection via Neural Architecture Search and Attention [pdf] [code] Inpainting detection
2021 WWW 2021 Progressive Semantic Reasoning for Image Inpainting [pdf] [code]
2021 ICCV 2021 Occlusion-Aware Video Object Inpainting [pdf] Video
2021 ICCV 2021 Internal Video Inpainting by Implicit Long-range Propagation [pdf] [code] Video
2021 ICCV 2021 Distillation-Guided Image Inpainting [pdf]
2021 ICCV 2021 Frequency-Aware Spatiotemporal Transformers for Video Inpainting Detection [pdf] Inpainting detection
2021 ICCV 2021 SLIDE: Single Image 3D Photography With Soft Layering and Depth-Aware Inpainting [pdf] [project]
2021 ICCV 2021 FuseFormer: Fusing Fine-Grained Information in Transformers for Video Inpainting [pdf] [code] Video
2021 ICCV 2021 WaveFill: A Wavelet-Based Generation Network for Image Inpainting [pdf]
2021 ICCV 2021 CR-Fill: Generative Image Inpainting With Auxiliary Contextual Reconstruction [pdf] [code]
2021 ICCV 2021 Learning a Sketch Tensor Space for Image Inpainting of Man-Made Scenes [pdf] [project]
2021 ICCV 2021 Parallel Multi-Resolution Fusion Network for Image Inpainting [pdf]
2021 ICCV 2021 Flow-Guided Video Inpainting With Scene Templates [pdf]
2021 ICCV 2021 High-Fidelity Pluralistic Image Completion With Transformers [pdf] [project]
2021 ICCV 2021 Learning High-Fidelity Face Texture Completion Without Complete Face Texture [pdf] Face
2021 WACV 2022 Resolution-robust Large Mask Inpainting with Fourier Convolutions [pdf] [code]
2022 CVPR 2022 Dual-path Image Inpainting with Auxiliary GAN Inversion
2022 CVPR 2022 MAT: Mask-Aware Transformer for Large Hole Image Inpainting [pdf] [code]
2022 CVPR 2022 Incremental Transformer Structure Enhanced Image Inpainting with Masking Positional Encoding [pdf] [code]
2022 CVPR 2022 Reduce Information Loss in Transformers for Pluralistic Image Inpainting
2022 CVPR 2022 MISF: Multi-level Interactive Siamese Filtering for High-Fidelity Image Inpainting [pdf] [code]
2022 CVPR 2022 RePaint: Inpainting using Denoising Diffusion Probabilistic Models [pdf] [code]
2022 CVPR 2022 DLFormer:Discrete Latent Transformer for Video Inpainting Video
2022 CVPR 2022 The DEVIL is in the Details: A Diagnostic Evaluation Benchmark for Video Inpainting [pdf] [code] Video
2022 CVPR 2022 Towards An End-to-End Framework for Flow-Guided Video Inpainting [pdf] [code] Video
2022 CVPR 2022 Inertia-Guided Flow Completion and Style Fusion for Video Inpainting Video
2022 CVPR 2022 DLFormer:Discrete Latent Transformer for Video Inpainting Video

论文研读

今天这篇论文的主角是Image Inpainting via Generative Multi-column Convolutional Neural Networks,相关代码可以参考我的github:Code。

这篇论文提出了generative multi-column network(GMCNN)。摘要如下:

In this paper, we propose a generative multi-column network for image inpainting. This network synthesizes different image components in a parallel manner within one stage. To better characterize global structures, we design a confidence-driven reconstruction loss while an implicit diversified MRF regularization is adopted to enhance local details. The multi-column network combined with the reconstruction and MRF loss propagates local and global information derived from context to the target inpainting regions. Extensive experiments on challenging street view, face, natural objects and scenes manifest that our method produces visual compelling results even without previously common post-processing.

图像复原的概念:

Image inpainting (also known as image completion) aims to estimate suitable pixel information to fill holes in images. It serves various applications such as object removal, image restoration, image denoising, to name a few.

经典的图像复原方法主要定位以下三个重要的问题:

  1. 提取合适的特征分析patch之间的相似性
  2. 找到近邻的patches
  3. 汇总辅助信息
    在这里插入图片描述
    ImageInpaint:图像修复_第1张图片
    ImageInpaint:图像修复_第2张图片
    ImageInpaint:图像修复_第3张图片

方法

ImageInpaint:图像修复_第4张图片
ImageInpaint:图像修复_第5张图片
该网络主要有三部分:

  1. a generator to produce results
  2. global and local discriminators for adversarial training
  3. a pretrained VGG network to calculate ID-MRF loss

网络结构

  • generator
    ImageInpaint:图像修复_第6张图片

  • ID-MRF Regularization
    马尔科夫随机场项
    ImageInpaint:图像修复_第7张图片
    ImageInpaint:图像修复_第8张图片

ImageInpaint:图像修复_第9张图片

  • Information Fusion
    ImageInpaint:图像修复_第10张图片
    ImageInpaint:图像修复_第11张图片

参考

  1. Image Inpainting
  2. F. Agostinelli, M. R. Anderson, and H. Lee. Adaptive multi-column deep neural networks with application to robust image denoising. In NIPS, pages 1493–1501, 2013
  3. D. Ciregan, U. Meier, and J. Schmidhuber. Multi-column deep neural networks for image classification. In CVPR, pages 3642–3649. IEEE, 2012
  4. Y. Zhang, D. Zhou, S. Chen, S. Gao, and Y. Ma. Single-image crowd counting via multi-column convolutional neural network. In CVPR, pages 589–597, 2016.

你可能感兴趣的:(计算机视觉,计算机视觉,深度学习,图像修复)