ImageInpaint：图像修复
文章目录
- 前言
- 传统方法
- 深度学习
- 论文研读
-
- 方法
-
- 网络结构
- 参考
前言
传统方法
| Year | Proceeding | Title | Tag |
|---|---|---|---|
| 2000 | SIGGRAPH 2000 | Image Inpainting [pdf] | Diffusion-based |
| 2001 | TIP 2001 | Filling-in by joint interpolation of vector fields and gray levels [pdf] | Diffusion-based |
| 2001 | CVPR 2001 | Navier-stokes, fluid dynamics, and image and video inpainting [pdf] | |
| 2001 | SIGGRAPH 2001 | Image Quilting for Texture Synthesis and Transfer [pdf] | |
| 2001 | SIGGRAPH 2001 | Synthesizing Natural Textures [pdf] | |
| 2002 | EJAM 2002 | Digital inpainting based on the mumford–shah–euler image model [pdf] | Diffusion-based |
| 2003 | CVPR 2003 | Object removal by exemplar-based inpainting [pdf] | |
| 2003 | TIP 2003 | Simultaneous structure and texture image inpainting [pdf] | Diffusion-based |
| 2003 | TIP 2003 | Structure and Texture Filling-In of Missing Image Blocks in Wireless Transmission and Compression Applications [pdf] | |
| 2003 | ICCV 2003 | Learning How to Inpaint from Global Image Statistics [pdf] | Diffusion-based |
| 2003 | TOG 2003 | Fragment-based image completion [pdf] | Patch-based |
| 2004 | TIP 2004 | Region Filling and Object Removal by Exemplar-Based Image Inpainting [pdf] | Patch-based; Inpainting order |
| 2004 | TPAMI 2004 | Space-Time Video Completion [pdf] | |
| 2005 | SIGGRAPH 2005 | Image Completion with Structure Propagation [pdf] | Patch-based |
| 2006 | ISCS 2006 | Image Compression with Structure Aware Inpainting [pdf] | |
| 2007 | TOG 2007 | Scene completion using millions of photographs [pdf] | |
| 2007 | CSVT 2007 | Image Compression With Edge-Based Inpainting [pdf] | Diffusion-based |
| 2008 | CVPR 2008 | Summarizing Visual Data Using Bidirectional Similarity [pdf] | |
| 2009 | SIGGRAPH 2009 | PatchMatch: a randomized correspondence algorithm for structural image editing [pdf] | Patch-based |
| 2010 | TIP 2010 | Image inpainting by patch propagation using patch sparsity [pdf] | Patch-based |
| 2011 | FTCGV 2011 | Structured learning and prediction in computer vision [pdf] | |
| 2011 | ICIP 2011 | Examplar-based inpainting based on local geometry [pdf] | Inpainting order |
| 2012 | TOG 2012 | Combining inconsistent images using patch-based synthesis[pdf] | Patch-based |
| 2014 | TOG 2014 | Image completion using Planar structure guidance [pdf] | Patch-based |
| 2014 | TVCG 2014 | High-Quality Real-Time Video Inpainting with PixMix [pdf] | Video |
| 2014 | SIAM 2014 | Video inpainting of complex scenes [pdf] | Video |
| 2015 | TIP 2015 | Annihilating Filter-Based Low-Rank Hankel Matrix Approach for Image Inpainting [pdf] | |
| 2015 | TIP 2015 | Exemplar-Based Inpainting: Technical Review and New Heuristics for Better Geometric Reconstructions [pdf] | |
| 2016 | TOG 2016 | Temporally coherent completion of dynamic video [pdf] | Video |
深度学习
| Year | Proceeding | Title | Tag |
|---|---|---|---|
| 2012 | NIPS 2012 | Image denoising and inpainting with deep neural networks [pdf] | |
| 2014 | GCPR 2014 | Mask-specific inpainting with deep neural networks [pdf] | |
| 2014 | NIPS 2014 | Deep Convolutional Neural Network for Image Deconvolution [pdf] | |
| 2015 | NIPS 2015 | Shepard Convolutional Neural Networks [pdf] [code] | |
| 2016 | CVPR 2016 | Context Encoders: Feature Learning by Inpainting [pdf] [code] | |
| 2016 | SIGGRAPH 2016 | High-resolution multi-scale neural texture synthesis [pdf] | |
| 2017 | CVPR 2017 | Semantic image inpainting with deep generative models [pdf] [code] | |
| 2017 | CVPR 2017 | High-resolution image inpainting using multi-scale neural patch synthesis [pdf] [code] | |
| 2017 | CVPR 2017 | Generative Face Completion [pdf] [code] | |
| 2017 | SIGGRAPH 2017 | Globally and Locally Consistent Image Completion [pdf] [code] | |
| 2018 | CVPR 2018 | Generative Image Inpainting with Contextual Attention [pdf] [code] | |
| 2018 | CVPR 2018 | Natural and Effective Obfuscation by Head Inpainting [pdf] | |
| 2018 | CVPR 2018 | Eye In-Painting With Exemplar Generative Adversarial Networks [pdf] [code] | |
| 2018 | CVPR 2018 | UV-GAN: Adversarial Facial UV Map Completion for Pose-invariant Face Recognition [pdf] | |
| 2018 | CVPR 2018 | Disentangling Structure and Aesthetics for Style-aware Image Completion [pdf] | |
| 2018 | ECCV 2018 | Image Inpainting for Irregular Holes Using Partial Convolutions [pdf] [code] | |
| 2018 | ECCV 2018 | Contextual-based Image Inpainting: Infer, Match, and Translate [pdf] | |
| 2018 | ECCV 2018 | Shift-Net: Image Inpainting via Deep Feature Rearrangement [pdf] [code] | |
| 2018 | NIPS 2018 | Image Inpainting via Generative Multi-column Convolutional Neural Networks [pdf] [code] | |
| 2018 | TOG 2018 | Faceshop: Deep sketch-based face image editing [pdf] | |
| 2018 | ACM MM 2018 | Structural inpainting [pdf] | |
| 2018 | ACM MM 2018 | Semantic Image Inpainting with Progressive Generative Networks [pdf] [code] | |
| 2018 | BMVC 2018 | SPG-Net: Segmentation Prediction and Guidance Network for Image Inpainting [pdf] | |
| 2018 | BMVC 2018 | MC-GAN: Multi-conditional Generative Adversarial Network for Image Synthesi [pdf] [code] | |
| 2018 | ACCV 2018 | Face Completion iwht Semantic Knowledge and Collaborative Adversarial Learning [pdf] | |
| 2018 | ICASSP 2018 | Edge-Aware Context Encoder for Image Inpainting [pdf] | |
| 2018 | ICPR 2018 | Deep Structured Energy-Based Image Inpainting [pdf] [code] | |
| 2018 | AISTATS 2019 | Probabilistic Semantic Inpainting with Pixel Constrained CNNs [pdf] | |
| 2019 | ICRA 2019 | Empty Cities: Image Inpainting for a Dynamic-Object-Invariant Space [pdf] | |
| 2019 | AAAI 2019 | Video Inpainting by Jointly Learning Temporal Structure and Spatial Details [pdf] | Video |
| 2019 | CVPR 2019 | Pluralistic Image Completion [pdf] [code] [project] | |
| 2019 | CVPR 2019 | Deep Reinforcement Learning of Volume-guided Progressive View Inpainting for 3D Point Scene Completion from a Single Depth Image [pdf] | |
| 2019 | CVPR 2019 | Foreground-aware Image Inpainting [pdf] | |
| 2019 | CVPR 2019 | Privacy Protection in Street-View Panoramas using Depth and Multi-View Imagery [pdf] | |
| 2019 | CVPR 2019 | Learning Pyramid-Context Encoder Network for High-Quality Image Inpainting [pdf] [code] | |
| 2019 | CVPR 2019 | Deep Flow-Guided Video Inpainting [pdf] [project] | Video |
| 2019 | CVPR 2019 | Deep video inapinting [pdf] | Video |
| 2019 | CVPRW 2019 | VORNet: Spatio-temporally Consistent Video Inpainting for Object Removal [pdf] | Video |
| 2019 | TNNLS 2019 | PEPSI++: Fast and Lightweight Network for Image Inpainting [pdf] | |
| 2019 | IJCAI 2019 | MUSICAL: Multi-Scale Image Contextual Attention Learning for Inpainting [pdf] | |
| 2019 | IJCAI 2019 | Generative Image Inpainting with Submanifold Alignment [pdf] | |
| 2019 | ACM MM 2019 | Progressive Image Inpainting with Full-Resolution Residual Network [pdf] [code] | |
| 2019 | ACM MM 2019 | Deep Fusion Network for Image Completion [pdf] [code] | |
| 2019 | ACM MM 2019 | GAIN: Gradient Augmented Inpainting Network for Irregular Holes [pdf] | |
| 2019 | ACM MM 2019 | Single-shot Semantic Image Inpainting with Densely Connected Generative Networks [pdf] | |
| 2019 | ICCVW 2019 | EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning [pdf] [code] | |
| 2019 | ICCV 2019 | Coherent Semantic Attention for Image Inpainting [pdf] [code] | |
| 2019 | ICCV 2019 | StructureFlow: Image Inpainting via Structure-aware Appearance Flow [pdf] [code] | |
| 2019 | ICCV 2019 | Progressive Reconstruction of Visual Structure for Image Inpainting [pdf] [code] | |
| 2019 | ICCV 2019 | Localization of Deep Inpainting Using High-Pass Fully Convolutional Network [pdf] | |
| 2019 | ICCV 2019 | Image Inpainting with Learnable Bidirectional Attention Maps [pdf] [code] | |
| 2019 | ICCV 2019 | Free-Form Image Inpainting with Gated Convolution [pdf] [project] | |
| 2019 | ICCV 2019 | FiNet: Compatible and Diverse Fashion Image Inpainting [pdf] | Fashion |
| 2019 | ICCV 2019 | SC-FEGAN: Face Editing Generative Adversarial Network with User’s Sketch and Color [pdf] [code] | Face |
| 2019 | ICCV 2019 | Human Motion Prediction via Spatio-Temporal Inpainting [pdf] [code] | Motion |
| 2019 | ICCV 2019 | Copy-and-Paste Networks for Deep Video Inpainting [pdf] [code] | Video |
| 2019 | ICCV 2019 | Onion-Peel Networks for Deep Video Completion [pdf] [code] | Video |
| 2019 | ICCV 2019 | Free-form Video Inpainting with 3D Gated Convolution and Temporal PatchGAN [pdf] [code] | Video |
| 2019 | ICCV 2019 | An Internal Learning Approach to Video Inpainting [pdf] | Video |
| 2019 | ICCV 2019 | Vision-Infused Deep Audio Inpainting [pdf] [code] | Audio |
| 2019 | AAAI 2020 | Region Normalization for Image Inpainting [pdf] [code] | |
| 2019 | AAAI 2020 | Learning to Incorporate Structure Knowledge for Image Inpainting [pdf] [code] | |
| 2020 | CVPR 2020 | Prior Guided GAN Based Semantic Inpainting [pdf] | |
| 2020 | CVPR 2020 | UCTGAN: Diverse Image Inpainting based on Unsupervised Cross-Space Translation [pdf] | |
| 2020 | CVPR 2020 | Recurrent Feature Reasoning for Image Inpainting [pdf] [code] | |
| 2020 | CVPR 2020 | Contextual Residual Aggregation for Ultra High-Resolution Image Inpainting [pdf] [code] | |
| 2020 | CVPR 2020 | 3D Photography using Context-aware Layered Depth Inpainting [pdf] [code] | |
| 2020 | CVPR 2020 | Learning Oracle Attention for High-fidelity Face Completion [pdf] | |
| 2020 | ECCV 2020 | Rethinking Image Inpainting via a Mutual Encoder-Decoder with Feature Equalizations [pdf] [code] | |
| 2020 | ECCV 2020 | Short-Term and Long-Term Context Aggregation Network for Video Inpainting | Video |
| 2020 | ECCV 2020 | Learning Object Placement by Inpainting for Compositional Data Augmentation | |
| 2020 | ECCV 2020 | Learning Joint Spatial-Temporal Transformations for Video Inpainting [pdf] [code] | Video |
| 2020 | ECCV 2020 | High-Resolution Image Inpainting with Iterative Confidence Feedback and Guided Upsampling [pdf] | |
| 2020 | ECCV 2020 | DVI: Depth Guided Video Inpainting for Autonomous Driving [pdf] [code] | Video |
| 2020 | ECCV 2020 | VCNet: A Robust Approach to Blind Image Inpainting [pdf] | |
| 2020 | ECCV 2020 | Guidance and Evaluation: Semantic-Aware Image Inpainting for Mixed Scenes [pdf] | |
| 2021 | WACV 2021 | Hyperrealistic Image Inpainting with Hypergraphs [pdf] [code] | |
| 2021 | CVPR 2021 | Generating Diverse Structure for Image Inpainting With Hierarchical VQ-VAE [pdf] [code] | |
| 2021 | CVPR 2021 | Image Inpainting with External-internal Learning and Monochromic Bottleneck [pdf] [code] | |
| 2021 | CVPR 2021 | PD-GAN: Probabilistic Diverse GAN for Image Inpainting [pdf] [code] | |
| 2021 | CVPR 2021 | Image Inpainting Guided by Coherence Priors of Semantics and Textures [pdf] | |
| 2021 | CVPR 2021 | FaceInpainter: High Fidelity Face Adaptation to Heterogeneous Domains [pdf] | |
| 2021 | CVPR 2021 | TransFill: Reference-guided Image Inpainting by Merging Multiple Color and Spatial Transformations [pdf] [code] | |
| 2021 | CVPR 2021 | Prior Based Human Completion [pdf] | Human completion |
| 2021 | IJCAI 2021 | Context-Aware Image Inpainting with Learned Semantic Priors [pdf] [code] | |
| 2021 | IJCAI 2021 | Noise Doesn’t Lie: Towards Universal Detection of Deep Inpainting [pdf] | Inpainting detection |
| 2021 | TCSVT 2021 | IID-Net: Image Inpainting Detection via Neural Architecture Search and Attention [pdf] [code] | Inpainting detection |
| 2021 | WWW 2021 | Progressive Semantic Reasoning for Image Inpainting [pdf] [code] | |
| 2021 | ICCV 2021 | Occlusion-Aware Video Object Inpainting [pdf] | Video |
| 2021 | ICCV 2021 | Internal Video Inpainting by Implicit Long-range Propagation [pdf] [code] | Video |
| 2021 | ICCV 2021 | Distillation-Guided Image Inpainting [pdf] | |
| 2021 | ICCV 2021 | Frequency-Aware Spatiotemporal Transformers for Video Inpainting Detection [pdf] | Inpainting detection |
| 2021 | ICCV 2021 | SLIDE: Single Image 3D Photography With Soft Layering and Depth-Aware Inpainting [pdf] [project] | |
| 2021 | ICCV 2021 | FuseFormer: Fusing Fine-Grained Information in Transformers for Video Inpainting [pdf] [code] | Video |
| 2021 | ICCV 2021 | WaveFill: A Wavelet-Based Generation Network for Image Inpainting [pdf] | |
| 2021 | ICCV 2021 | CR-Fill: Generative Image Inpainting With Auxiliary Contextual Reconstruction [pdf] [code] | |
| 2021 | ICCV 2021 | Learning a Sketch Tensor Space for Image Inpainting of Man-Made Scenes [pdf] [project] | |
| 2021 | ICCV 2021 | Parallel Multi-Resolution Fusion Network for Image Inpainting [pdf] | |
| 2021 | ICCV 2021 | Flow-Guided Video Inpainting With Scene Templates [pdf] | |
| 2021 | ICCV 2021 | High-Fidelity Pluralistic Image Completion With Transformers [pdf] [project] | |
| 2021 | ICCV 2021 | Learning High-Fidelity Face Texture Completion Without Complete Face Texture [pdf] | Face |
| 2021 | WACV 2022 | Resolution-robust Large Mask Inpainting with Fourier Convolutions [pdf] [code] | |
| 2022 | CVPR 2022 | Dual-path Image Inpainting with Auxiliary GAN Inversion | |
| 2022 | CVPR 2022 | MAT: Mask-Aware Transformer for Large Hole Image Inpainting [pdf] [code] | |
| 2022 | CVPR 2022 | Incremental Transformer Structure Enhanced Image Inpainting with Masking Positional Encoding [pdf] [code] | |
| 2022 | CVPR 2022 | Reduce Information Loss in Transformers for Pluralistic Image Inpainting | |
| 2022 | CVPR 2022 | MISF: Multi-level Interactive Siamese Filtering for High-Fidelity Image Inpainting [pdf] [code] | |
| 2022 | CVPR 2022 | RePaint: Inpainting using Denoising Diffusion Probabilistic Models [pdf] [code] | |
| 2022 | CVPR 2022 | DLFormer:Discrete Latent Transformer for Video Inpainting | Video |
| 2022 | CVPR 2022 | The DEVIL is in the Details: A Diagnostic Evaluation Benchmark for Video Inpainting [pdf] [code] | Video |
| 2022 | CVPR 2022 | Towards An End-to-End Framework for Flow-Guided Video Inpainting [pdf] [code] | Video |
| 2022 | CVPR 2022 | Inertia-Guided Flow Completion and Style Fusion for Video Inpainting | Video |
| 2022 | CVPR 2022 | DLFormer:Discrete Latent Transformer for Video Inpainting | Video |
论文研读
今天这篇论文的主角是Image Inpainting via Generative Multi-column Convolutional Neural Networks，相关代码可以参考我的github：Code。
这篇论文提出了generative multi-column network（GMCNN）。摘要如下：
In this paper, we propose a generative multi-column network for image inpainting. This network synthesizes different image components in a parallel manner within one stage. To better characterize global structures, we design a confidence-driven reconstruction loss while an implicit diversified MRF regularization is adopted to enhance local details. The multi-column network combined with the reconstruction and MRF loss propagates local and global information derived from context to the target inpainting regions. Extensive experiments on challenging street view, face, natural objects and scenes manifest that our method produces visual compelling results even without previously common post-processing.
图像复原的概念：
Image inpainting (also known as image completion) aims to estimate suitable pixel information to fill holes in images. It serves various applications such as object removal, image restoration, image denoising, to name a few.
经典的图像复原方法主要定位以下三个重要的问题：
- 提取合适的特征分析patch之间的相似性
- 找到近邻的patches
- 汇总辅助信息
方法
该网络主要有三部分：
- a generator to produce results
- global and local discriminators for adversarial training
- a pretrained VGG network to calculate ID-MRF loss
网络结构
-
generator
-
ID-MRF Regularization
马尔科夫随机场项
- Information Fusion
参考
- Image Inpainting
- F. Agostinelli, M. R. Anderson, and H. Lee. Adaptive multi-column deep neural networks with application to robust image denoising. In NIPS, pages 1493–1501, 2013
- D. Ciregan, U. Meier, and J. Schmidhuber. Multi-column deep neural networks for image classification. In CVPR, pages 3642–3649. IEEE, 2012
- Y. Zhang, D. Zhou, S. Chen, S. Gao, and Y. Ma. Single-image crowd counting via multi-column convolutional neural network. In CVPR, pages 589–597, 2016.