【CV】图像恢复（降噪/超分/去雾/去雨/去模糊）顶会论文汇总

---

A survey of deep learning approaches to image restoration

论文名称：A survey of deep learning approaches to image restoration

论文下载：https://www.sciencedirect.com/science/article/pii/S0925231222002089?via%3Dihub

论文年份：Neurocomputing 2022

In this paper, we present an extensive review on deep learning methods for image restoration tasks. Deep learning techniques, led by convolutional neural networks, have received a great deal of attention in almost all areas of image processing, especially in image classification. However, image restoration is a fundamental and challenging topic and plays significant roles in image processing, understanding and representation. It typically addresses image deblurring, denoising, dehazing and super-resolution. There are substantial differences in the approaches and mechanisms in deep learning methods for image restoration. Discriminative learning based methods are able to deal with issues of learning a restoration mapping function effectively, while optimisation models based methods can further enhance the performance with certain learning constraints. In this paper, we offer a comparative study of deep learning techniques in image denoising, deblurring, dehazing, and super-resolution, and summarise the principles involved in these tasks from various supervised deep network architectures, residual or skip connection and receptive field to unsupervised autoencoder mechanisms. Image quality criteria are also reviewed and their roles in image restoration are assessed. Based on our analysis, we further present an efficient network for deblurring and a couple of multi-objective training functions for super-resolution restoration tasks. The proposed methods are compared extensively with the state-of-the-art methods with both quantitative and qualitative analyses. Finally, we point out potential challenges and directions for future research.

在本文中，我们对用于图像恢复任务的深度学习方法进行了广泛的回顾。以卷积神经网络为首的深度学习技术在几乎所有图像处理领域，尤其是图像分类领域，都受到了广泛关注。然而，图像恢复是一个基本且具有挑战性的课题，在图像处理、理解和表示中起着重要作用。它通常处理图像去模糊、去噪、去雾和超分辨率。用于图像恢复的深度学习方法的方法和机制存在很大差异。基于判别学习的方法能够有效地处理学习恢复映射函数的问题，而基于优化模型的方法可以在一定的学习约束下进一步提高性能。在本文中，我们对图像去噪、去模糊、去雾和超分辨率中的深度学习技术进行了比较研究，并总结了这些任务所涉及的原理，从各种有监督的深度网络架构、残差或跳过连接和感受野到无监督自动编码器机制。还审查了图像质量标准，并评估了它们在图像恢复中的作用。基于我们的分析，我们进一步提出了一个有效的去模糊网络和几个用于超分辨率恢复任务的多目标训练函数。所提出的方法与定量和定性分析的最新方法进行了广泛的比较。最后，我们指出了未来研究的潜在挑战和方向。

Denoising Prior Driven Deep Neural Network for Image Restoration

论文名称：Denoising Prior Driven Deep Neural Network for Image Restoration

论文下载：https://ieeexplore.ieee.org/abstract/document/8481558

论文年份：TPAMI 2019

论文被引：221（2022/05/04）

论文代码：https://github.com/WeishengDong/DPDNN_PyTorch

Deep neural networks (DNNs) have shown very promising results for various image restoration (IR) tasks. However, the design of network architectures remains a major challenging for achieving further improvements. While most existing DNN-based methods solve the IR problems by directly mapping low quality images to desirable high-quality images, the observation models characterizing the image degradation processes have been largely ignored. In this paper, we first propose a denoising-based IR algorithm, whose iterative steps can be computed efficiently. Then, the iterative process is unfolded into a deep neural network, which is composed of multiple denoisers modules interleaved with back-projection (BP) modules that ensure the observation consistencies. A convolutional neural network (CNN) based denoiser that can exploit the multiscale redundancies of natural images is proposed. As such, the proposed network not only exploits the powerful denoising ability of DNNs, but also leverages the prior of the observation model. Through end-to-end training, both the denoisers and the BP modules can be jointly optimized. Experimental results on several IR tasks, e.g., image denoisig, super-resolution and deblurring show that the proposed method can lead to very competitive and often state-of-the-art results on several IR tasks, including image denoising, deblurring and super-resolution.

深度神经网络 (DNN) 已在各种 图像恢复 (image restoration, IR) 任务中显示出非常有希望的结果。然而，网络架构的设计仍然是实现进一步改进的主要挑战。虽然大多数现有的基于 DNN 的方法通过将低质量图像直接映射到所需的高质量图像来解决 IR 问题，但 表征图像退化过程 (image degradation processes) 的观察模型 在很大程度上被忽略了。在本文中，我们首先提出了一种 基于去噪的 IR 算法，该算法的迭代步骤可以有效地计算。然后，将迭代过程展开为一个深度神经网络，该网络由多个降噪模块和反投影 (back-projection, BP) 模块交织而成，以确保观测一致性。提出了一种基于卷积神经网络 (CNN) 的 去噪 (denoising) 方法，可以利用自然图像的多尺度冗余。因此，所提出的网络不仅利用了 DNN 强大的去噪能力，而且 利用了观察模型的先验。通过端到端的训练，降噪器和BP模块都可以联合优化。在图像去噪、超分辨率和去模糊等几个 IR 任务上的实验结果表明，所提出的方法可以产生了非常有竞争力且通常是最先进的结果。

Deep Gaussian Scale Mixture Prior for Spectral Compressive Imaging

论文名称：Deep Gaussian Scale Mixture Prior for Spectral Compressive Imaging

论文下载：link

论文年份：CVPR 2021

论文被引：19（2022/05/04）

论文代码：https://github.com/TaoHuang95/DGSMP

In coded aperture snapshot spectral imaging (CASSI) system, the real-world hyperspectral image (HSI) can be reconstructed from the captured compressive image in a snapshot. Model-based HSI reconstruction methods employed hand-crafted priors to solve the reconstruction problem, but most of which achieved limited success due to the poor representation capability of these hand-crafted priors. Deep learning based methods learning the mappings between the compressive images and the HSIs directly achieved much better results. Yet, it is nontrivial to design a powerful deep network heuristically for achieving satisfied results. In this paper, we propose a novel HSI reconstruction method based on the Maximum a Posterior (MAP) estimation framework using learned Gaussian Scale Mixture (GSM) prior. Different from existing GSM models using hand-crafted scale priors (e.g., the Jeffrey’s prior), we propose to learn the scale prior through a deep convolutional neural network (DCNN). Furthermore, we also propose to estimate the local means of the GSM models by the DCNN. All the parameters of the MAP estimation algorithm and the DCNN parameters are jointly optimized through end-to-end training. Extensive experimental results on both synthetic and real datasets demonstrate that the proposed method outperforms existing state-of-the-art methods.

在编码孔径快照光谱成像 (coded aperture snapshot spectral imaging ，CASSI) 系统中，可以从快照中捕获的压缩图像重建真实世界的高光谱图像 (hyperspectral image，HSI)。基于模型的 HSI 重建方法采用手工制作的先验来解决重建问题，但由于这些手工制作的先验的表示能力差，大多数方法取得的成功有限。基于深度学习的方法学习压缩图像和 HSI 之间的映射直接取得了更好的结果。然而，启发式地设计强大的深度网络以获得满意的结果并非易事。在本文中，我们提出了一种新的 HSI 重建方法，该方法基于使用学习的高斯尺度混合 (Gaussian Scale Mixture ，GSM) 先验的最大后验 (MAP) 估计框架。与使用手工制作的尺度先验（例如，Jeffrey 先验）的现有 GSM 模型不同，我们建议通过深度卷积神经网络（DCNN）来学习尺度先验。此外，我们还建议通过 DCNN 估计 GSM 模型的局部均值。 MAP估计算法的所有参数和DCNN参数通过端到端训练联合优化。在合成数据集和真实数据集上的大量实验结果表明，所提出的方法优于现有的最先进方法。

Image-to-Image MLP-mixer for Image Reconstruction

论文名称：Image-to-Image MLP-mixer for Image Reconstruction

论文下载：https://arxiv.org/abs/2202.02018

论文年份：ICLR 2022 （在审）

论文被引：

论文代码：

Neural networks are highly effective tools for image reconstruction problems such as denoising and compressive sensing. To date, neural networks for image reconstruction are almost exclusively convolutional. The most popular architecture is the U-Net, a convolutional network with a multi-resolution architecture. In this work, we show that a simple network based on the multi-layer perceptron (MLP)-mixer enables state-of-the art image reconstruction performance without convolutions and without a multi-resolution architecture, provided that the training set and the size of the network are moderately large. Similar to the original MLP-mixer, the image-to-image MLP-mixer is based exclusively on MLPs operating on linearly-transformed image patches. Contrary to the original MLP-mixer, we incorporate structure by retaining the relative positions of the image patches. This imposes an inductive bias towards natural images which enables the image-to-image MLP-mixer to learn to denoise images based on fewer examples than the original MLP-mixer. Moreover, the image-to-image MLP-mixer requires fewer parameters to achieve the same denoising performance than the U-Net and its parameters scale linearly in the image resolution instead of quadratically as for the original MLP-mixer. If trained on a moderate amount of examples for denoising, the image-to-image MLP-mixer outperforms the U-Net by a slight margin. It also outperforms the vision transformer tailored for image reconstruction and classical un-trained methods such as BM3D, making it a very effective tool for image reconstruction problems.

神经网络是解决图像重建问题（例如去噪和压缩感知）的高效工具。迄今为止，用于图像重建的神经网络几乎完全是卷积的。最流行的架构是 U-Net，一种具有多分辨率架构的卷积网络。在这项工作中，我们展示了一个基于多层感知器 (MLP)-mixer 的简单网络可以在没有卷积和多分辨率架构的情况下实现最先进的图像重建性能，前提是训练集和大小网络规模适中。与原始 MLP-mixer类似，图像到图像 MLP-mixer仅基于在线性变换图像块上运行的 MLP。与原始的 MLP-mixer相反，我们通过保留图像块的相对位置来合并结构。这对自然图像施加了一种归纳偏差，使图像到图像的 MLP-mixer能够基于比原始 MLP-mixer更少的示例来学习对图像进行去噪。此外，图像到图像的 MLP-mixer需要更少的参数来实现与 U-Net 相同的去噪性能，并且其参数在图像分辨率中线性缩放，而不是原始 MLP-mixer的二次方。如果在中等数量的去噪示例上进行训练，图像到图像 MLP-mixer的性能会略微优于 U-Net。它还优于为图像重建和经典未经训练的方法（如 BM3D）量身定制的视觉 Transformer，使其成为解决图像重建问题的非常有效的工具。

SwinIR: Image Restoration Using Swin Transformer

论文名称：SwinIR: Image Restoration Using Swin Transformer

论文下载：https://arxiv.org/abs/2108.10257

论文年份：ICCV 2021

论文被引：117（2022/05/04）

论文代码：https://github.com/JingyunLiang/SwinIR

Image restoration is a long-standing low-level vision problem that aims to restore high-quality images from lowquality images (e.g., downscaled, noisy and compressed images). While state-of-the-art image restoration methods are based on convolutional neural networks, few attempts have been made with Transformers which show impressive performance on high-level vision tasks. In this paper , we propose a strong baseline model SwinIR for image restoration based on the Swin Transformer . SwinIR consists of three parts: shallow feature extraction, deep feature extraction and high-quality image reconstruction. In particular , the deep feature extraction module is composed of several residual Swin Transformer blocks (RSTB), each of which has several Swin Transformer layers together with a residual connection. We conduct experiments on three representative tasks: image super-resolution (including classical, lightweight and real-world image super-resolution), image denoising (including grayscale and color image denoising) and JPEG compression artifact reduction. Experimental results demonstrate that SwinIR outperforms state-of-the-art methods on different tasks by up to 0.14∼0.45dB, while the total number of parameters can be reduced by up to 67%.

图像恢复是一个长期存在的低级视觉问题，旨在从低质量图像（例如，缩小、噪声和压缩图像）中恢复高质量图像。虽然最先进的图像恢复方法是基于卷积神经网络的，但很少有人尝试使用 Transformer 进行高级视觉任务的表现令人印象深刻。在本文中，我们提出了一种基于 Swin Transformer 的强大基线模型 SwinIR 用于图像恢复。 SwinIR 由浅层特征提取、深层特征提取和高质量图像重建三部分组成。特别是，深度特征提取模块由几个残差 Swin Transformer 块 (RSTB) 组成，每个残差块都有几个 Swin Transformer 层和一个残差连接。我们对三个具有代表性的任务进行了实验：图像超分辨率（包括经典、轻量级和真实世界的图像超分辨率），图像去噪（包括灰度和彩色图像去噪）和 JPEG 压缩伪影减少（JPEG compression artifact reduction）。实验结果表明，SwinIR 在不同任务上的性能优于最先进的方法高达 0.14∼0.45dB，而参数总数最多可减少 67%。

Uformer: A General U-Shaped Transformer for Image Restoration

论文名称：Uformer: A General U-Shaped Transformer for Image Restoration

论文下载：https://arxiv.org/abs/2106.03106

论文年份：CVPR 2022

论文被引：55（2022/05/04）

论文代码：https://github.com/ZhendongWang6/Uformer

In this paper , we present Uformer , an effective and efficient Transformer-based architecture for image restoration, in which we build a hierarchical encoder-decoder network using the Transformer block. In Uformer , there are two core designs. First, we introduce a novel locally-enhanced window (LeWin) Transformer block, which performs nonoverlapping window-based self-attention instead of global self-attention. It significantly reduces the computational complexity on high resolution feature map while capturing local context. Second, we propose a learnable multi-scale restoration modulator in the form of a multi-scale spatial bias to adjust features in multiple layers of the Uformer decoder . Our modulator demonstrates superior capability for restoring details for various image restoration tasks while introducing marginal extra parameters and computational cost. Powered by these two designs, Uformer enjoys a high capability for capturing both local and global dependencies for image restoration. To evaluate our approach, extensive experiments are conducted on several image restoration tasks, including image denoising, motion deblurring, defocus deblurring and deraining. Without bells and whistles, our Uformer achieves superior or comparable performance compared with the state-of-the-art algorithms.

在本文中，我们提出了 Uformer，这是一种有效且高效的基于 Transformer 的图像恢复架构，其中我们使用 Transformer 模块构建了一个分层的编码器-解码器网络。在 Uformer 中，有两种核心设计。首先，我们介绍了一种新颖的局部增强窗口 (LeWin) Transformer 块，它执行基于非重叠窗口的自注意而不是全局自注意。它在捕获局部上下文的同时显着降低了高分辨率特征图的计算复杂度。其次，我们提出了一种可学习的多尺度恢复调制器，以多尺度空间偏差的形式来调整 Uformer 解码器多层中的特征。我们的调制器展示了为各种图像恢复任务恢复细节的卓越能力，同时引入了边际额外参数和计算成本。在这两种设计的支持下，Uformer 具有捕获局部和全局依赖项以进行图像恢复的强大能力。为了评估我们的方法，我们对几个图像恢复任务进行了广泛的实验，包括图像去噪、运动去模糊、散焦去模糊和去雨。与最先进的算法相比，我们的 Uformer 无需花里胡哨，可实现卓越或可比的性能。

Restormer: Efficient Transformer for High-Resolution Image Restoration

论文名称：Restormer: Efficient Transformer for High-Resolution Image Restoration

论文下载：https://arxiv.org/abs/2111.09881

论文年份：CVPR 2022

论文被引：20（2022/05/04）

论文代码：https://github.com/swz30/Restormer

Since convolutional neural networks (CNNs) perform well at learning generalizable image priors from largescale data, these models have been extensively applied to image restoration and related tasks. Recently, another class of neural architectures, Transformers, have shown significant performance gains on natural language and high-level vision tasks. While the Transformer model mitigates the shortcomings of CNNs (i.e., limited receptive field and inadaptability to input content), its computational complexity grows quadratically with the spatial resolution, therefore making it infeasible to apply to most image restoration tasks involving high-resolution images. In this work, we propose an efficient Transformer model by making several key designs in the building blocks (multi-head attention and feed-forward network) such that it can capture long-range pixel interactions, while still remaining applicable to large images. Our model, named Restoration Transformer (Restormer), achieves state-of-the-art results on several image restoration tasks, including image deraining, single-image motion deblurring, defocus deblurring (single-image and dual-pixel data), and image denoising (Gaussian grayscale/color denoising, and real image denoising).

由于卷积神经网络 (CNN) 在从大规模数据中学习可概括的图像先验方面表现良好，因此这些模型已广泛应用于图像恢复和相关任务。最近，另一类神经架构 Transformers 在自然语言和高级视觉任务上显示出显着的性能提升。虽然 Transformer 模型减轻了 CNN 的缺点（即有限的感受野和不适应输入内容），但其计算复杂度随空间分辨率呈二次方增长，因此无法应用于大多数涉及高分辨率图像的图像恢复任务。在这项工作中，我们通过在构建块（多头注意力和前馈网络）中进行几个关键设计来提出一个高效的 Transformer 模型，以便它可以捕获远程像素交互，同时仍然适用于大图像。我们的模型名为 Restoration Transformer (Restormer)，在多个图像恢复任务上实现了最先进的结果，包括图像去雨、单图像运动去模糊、散焦去模糊（单图像和双像素数据）和图像去噪（高斯灰度/彩色去噪和真实图像去噪）。

Pyramid Attention Networks for Image Restoration

论文名称：Pyramid Attention Networks for Image Restoration

论文下载：https://arxiv.org/abs/2004.13824

论文年份：2020

论文被引：60（2022/05/04）

论文代码：https://github.com/SHI-Labs/Pyramid-Attention-Networks

Self-similarity refers to the image prior widely used in image restoration algorithms that small but similar patterns tend to occur at different locations and scales. However, recent advanced deep convolutional neural network based methods for image restoration do not take full advantage of self-similarities by relying on self-attention neural modules that only process information at the same scale. To solve this problem, we present a novel Pyramid Attention module for image restoration, which captures long-range feature correspondences from a multi-scale feature pyramid. Inspired by the fact that corruptions, such as noise or compression artifacts, drop drastically at coarser image scales, our attention module is designed to be able to borrow clean signals from their “clean” correspondences at the coarser levels. The proposed pyramid attention module is a generic building block that can be flexibly integrated into various neural architectures. Its effectiveness is validated through extensive experiments on multiple image restoration tasks: image denoising, demosaicing, compression artifact reduction, and super resolution. Without any bells and whistles, our PANet (pyramid attention module with simple network backbones) can produce state-ofthe-art results with superior accuracy and visual quality.

自相似性是指图像先验广泛用于图像恢复算法中，小但相似的图案往往出现在不同的位置和尺度上。然而，最近用于图像恢复的先进的基于深度卷积神经网络的方法并没有通过依赖仅处理相同尺度信息的自注意神经模块来充分利用自相似性。为了解决这个问题，我们提出了一种用于图像恢复的新型 Pyramid Attention 模块，它从多尺度特征金字塔中捕获远程特征对应关系。受诸如噪声或压缩伪影之类的损坏在较粗的图像尺度上急剧下降这一事实的启发，我们的注意力模块旨在能够从较粗级别的“干净”对应关系中借用干净的信号。所提出的金字塔注意力模块是一个通用的构建块，可以灵活地集成到各种神经架构中。它的有效性通过对多个图像恢复任务的广泛实验得到验证：图像去噪、去马赛克、压缩伪影减少和超分辨率。无需任何花里胡哨，我们的 PANet（具有简单网络主干的金字塔注意力模块）可以产生具有卓越准确性和视觉质量的最先进的结果。

Residual Non-local Attention Networks for Image Restoration

论文名称：Residual Non-local Attention Networks for Image Restoration

论文下载：https://arxiv.org/abs/1903.10082

论文年份：ICLR 2019

论文被引：344（2022/05/04）

论文代码：https://github.com/yulunzhang/RNAN

In this paper, we propose a residual non-local attention network for high-quality image restoration. Without considering the uneven distribution of information in the corrupted images, previous methods are restricted by local convolutional operation and equal treatment of spatial- and channel-wise features. To address this issue, we design local and non-local attention blocks to extract features that capture the long-range dependencies between pixels and pay more attention to the challenging parts. Specifically, we design trunk branch and (non-)local mask branch in each (non-)local attention block. The trunk branch is used to extract hierarchical features. Local and non-local mask branches aim to adaptively rescale these hierarchical features with mixed attentions. The local mask branch concentrates on more local structures with convolutional operations, while non-local attention considers more about long-range dependencies in the whole feature map. Furthermore, we propose residual local and non-local attention learning to train the very deep network, which further enhance the representation ability of the network. Our proposed method can be generalized for various image restoration applications, such as image denoising, demosaicing, compression artifacts reduction, and super-resolution. Experiments demonstrate that our method obtains comparable or better results compared with recently leading methods quantitatively and visually.

在本文中，我们提出了一种用于高质量图像恢复的残差非局部注意力网络。在不考虑损坏图像中信息的不均匀分布的情况下，以前的方法受到局部卷积运算以及空间和通道特征的平等处理的限制。为了解决这个问题，我们设计了局部和非局部注意力块来提取特征，捕捉像素之间的长期依赖关系，并更加关注具有挑战性的部分。具体来说，我们在每个（非）局部注意力块中设计了主干分支和（非）局部掩码分支。主干分支用于提取层次特征。局部和非局部掩码分支旨在通过混合注意力自适应地重新缩放这些分层特征。局部掩码分支专注于更多具有卷积操作的局部结构，而非局部注意力更多地考虑整个特征图中的长期依赖关系。此外，我们提出了残差局部和非局部注意力学习来训练非常深的网络，这进一步增强了网络的表示能力。我们提出的方法可以推广到各种图像恢复应用，例如图像去噪、去马赛克、压缩伪影减少和超分辨率。实验表明，与最近在定量和视觉上领先的方法相比，我们的方法获得了可比或更好的结果。

Learning Enriched Features for Real Image Restoration and Enhancement

论文名称：Learning Enriched Features for Real Image Restoration and Enhancement

论文下载：https://arxiv.org/abs/2003.06792

论文年份：ECCV 2020

论文被引：97（2022/05/04）

论文代码：https://github.com/swz30/MIRNet

With the goal of recovering high-quality image content from its degraded version, image restoration enjoys numerous applications, such as in surveillance, computational photography, medical imaging, and remote sensing. Recently, convolutional neural networks (CNNs) have achieved dramatic improvements over conventional approaches for image restoration task. Existing CNN-based methods typically operate either on full-resolution or on progressively low-resolution representations. In the former case, spatially precise but contextually less robust results are achieved, while in the latter case, semantically reliable but spatially less accurate outputs are generated. In this paper, we present a novel architecture with the collective goals of maintaining spatiallyprecise high-resolution representations through the entire network, and receiving strong contextual information from the low-resolution representations. The core of our approach is a multi-scale residual block containing several key elements: (a) parallel multi-resolution convolution streams for extracting multi-scale features, (b) information exchange across the multi-resolution streams, © spatial and channel attention mechanisms for capturing contextual information, and (d) attention based multi-scale feature aggregation. In a nutshell, our approach learns an enriched set of features that combines contextual information from multiple scales, while simultaneously preserving the high-resolution spatial details. Extensive experiments on five real image benchmark datasets demonstrate that our method, named as MIRNet, achieves state-of-the-art results for a variety of image processing tasks, including image denoising, super-resolution and image enhancement.

为了从退化图像中恢复高质量的图像内容，图像恢复在监控、计算摄影、医学成像和遥感等领域有着广泛的应用。最近，卷积神经网络（CNN）在图像恢复任务的传统方法上取得了显着的进步。现有的基于 CNN 的方法通常在全分辨率或逐步低分辨率表示上运行。在前一种情况下，获得了空间精确但在上下文上不太稳健的结果，而在后一种情况下，生成了语义可靠但空间上不太准确的输出。在本文中，我们提出了一种新颖的架构，其共同目标是通过整个网络维护空间精确的高分辨率表示，并从低分辨率表示中接收强大的上下文信息。我们方法的核心是包含几个关键元素的多尺度残差块：（a）用于提取多尺度特征的并行多分辨率卷积流，（b）跨多分辨率流的信息交换，（c）空间和用于捕获上下文信息的通道注意机制，以及（d）基于注意力的多尺度特征聚合。简而言之，我们的方法学习了一组丰富的特征，这些特征结合了来自多个尺度的上下文信息，同时保留了高分辨率的空间细节。在五个真实图像基准数据集上进行的大量实验表明，我们的方法（称为 MIRNet）在图像去噪、超分辨率和图像增强等各种图像处理任务中取得了最先进的结果。

Multi-Stage Progressive Image Restoration

论文名称：Multi-Stage Progressive Image Restoration

论文下载：link

论文年份：CVPR 2021

论文被引：154（2022/05/04）

论文代码：https://github.com/swz30/MPRNet

Image restoration tasks demand a complex balance between spatial details and high-level contextualized information while recovering images. In this paper , we propose a novel synergistic design that can optimally balance these competing goals. Our main proposal is a multi-stage architecture, that progressively learns restoration functions for the degraded inputs, thereby breaking down the overall recovery process into more manageable steps. Specifically, our model first learns the contextualized features using encoder-decoder architectures and later combines them with a high-resolution branch that retains local information. At each stage, we introduce a novel per-pixel adaptive design that leverages in-situ supervised attention to reweight the local features. A key ingredient in such a multi-stage architecture is the information exchange between different stages. To this end, we propose a twofaceted approach where the information is not only exchanged sequentially from early to late stages, but lateral connections between feature processing blocks also exist to avoid any loss of information. The resulting tightly interlinked multi-stage architecture, named as MPRNet, delivers strong performance gains on ten datasets across a range of tasks including image deraining, deblurring, and denoising.

图像恢复任务需要在恢复图像时在空间细节和高级上下文信息之间取得复杂的平衡。在本文中，我们提出了一种新颖的协同设计，可以最佳地平衡这些竞争目标。我们的主要提议是一个多阶段架构，它逐步学习退化输入的恢复函数，从而将整个恢复过程分解为更易于管理的步骤。具体来说，我们的模型首先使用编码器-解码器架构学习上下文特征，然后将它们与保留局部信息的高分辨率分支相结合。在每个阶段，我们引入了一种新颖的每像素自适应设计，该设计利用原位监督注意力来重新加权局部特征。这种多阶段架构的一个关键因素是不同阶段之间的信息交换。为此，我们提出了一种双方面的方法，其中信息不仅从早期阶段到晚期阶段顺序交换，而且特征处理块之间也存在横向连接以避免任何信息丢失。由此产生的紧密互连的多阶段架构，称为 MPRNet，在包括图像去雨、去模糊和去噪在内的一系列任务中，在十个数据集上提供了强大的性能提升。

CycleISP: Real Image Restoration via Improved Data Synthesis

论文名称：CycleISP: Real Image Restoration via Improved Data Synthesis

论文下载：link

论文年份：CVPR 2020 Oral

论文被引：96（2022/05/04）

论文代码：https://github.com/swz30/CycleISP

The availability of large-scale datasets has helped unleash the true potential of deep convolutional neural networks (CNNs). However , for the single-image denoising problem, capturing a real dataset is an unacceptably expensive and cumbersome procedure. Consequently, image denoising algorithms are mostly developed and evaluated on synthetic data that is usually generated with a widespread assumption of additive white Gaussian noise (AWGN). While the CNNs achieve impressive results on these synthetic datasets, they do not perform well when applied on real camera images, as reported in recent benchmark datasets. This is mainly because the AWGN is not adequate for modeling the real camera noise which is signaldependent and heavily transformed by the camera imaging pipeline. In this paper , we present a framework that models camera imaging pipeline in forward and reverse directions. It allows us to produce any number of realistic image pairs for denoising both in RAW and sRGB spaces. By training a new image denoising network on realistic synthetic data, we achieve the state-of-the-art performance on real camera benchmark datasets. The parameters in our model are ∼5 times lesser than the previous best method for RAW denoising. Furthermore, we demonstrate that the proposed framework generalizes beyond image denoising problem e.g., for color matching in stereoscopic cinema.

大规模数据集的可用性有助于释放深度卷积神经网络 (CNN) 的真正潜力。然而，对于单图像去噪问题，捕获真实数据集是一个不可接受的昂贵且繁琐的过程。因此，图像去噪算法主要是在合成数据上开发和评估的，这些合成数据通常是在广泛假设加性高斯白噪声 (AWGN) 的情况下生成的。虽然 CNN 在这些合成数据集上取得了令人印象深刻的结果，但正如最近的基准数据集所报告的那样，它们在应用于真实相机图像时表现不佳。这主要是因为 AWGN 不足以模拟真实的相机噪声，这种噪声是信号相关的并且由相机成像管道进行大量转换。在本文中，我们提出了一个在正向和反向方向上对相机成像管道进行建模的框架。它允许我们生成任意数量的真实图像对，用于在 RAW 和 sRGB 空间中进行去噪。通过在真实的合成数据上训练一个新的图像去噪网络，我们在真实的相机基准数据集上实现了最先进的性能。我们模型中的参数比以前最好的 RAW 去噪方法小 5 倍。此外，我们证明了所提出的框架可以推广到图像去噪问题之外，例如，用于立体电影中的颜色匹配。

HINet: Half Instance Normalization Network for Image Restoration

论文名称：HINet: Half Instance Normalization Network for Image Restoration

论文下载：https://arxiv.org/abs/2105.06086

论文年份：CVPR 2021

论文被引：30（2022/05/04）

论文代码：https://github.com/megvii-model/HINet

In this paper , we explore the role of Instance Normalization in low-level vision tasks. Specifically, we present a novel block: Half Instance Normalization Block (HIN Block), to boost the performance of image restoration networks. Based on HIN Block, we design a simple and powerful multi-stage network named HINet, which consists of two subnetworks. With the help of HIN Block, HINet surpasses the state-of-the-art (SOTA) on various image restoration tasks. For image denoising, we exceed it 0.11dB and 0.28 dB in PSNR on SIDD dataset, with only 7.5% and 30% of its multiplier-accumulator operations (MACs), 6.8× and 2.9× speedup respectively. F or image deblurring, we get comparable performance with 22.5% of its MACs and 3.3× speedup on REDS and GoPro datasets. F or image deraining, we exceed it by 0.3 dB in PSNR on the average result of multiple datasets with 1.4× speedup. With HINet, we won the 1st place on the NTIRE 2021 Image Deblurring Challenge - Track2. JPEG Artifacts, with a PSNR of 29.70.

在本文中，我们探讨了实例归一化在低级视觉任务中的作用。具体来说，我们提出了一个新颖的块：半实例归一化块（HIN 块），以提高图像恢复网络的性能。基于 HIN Block，我们设计了一个简单而强大的多级网络 HINet，它由两个子网络组成。在 HIN Block 的帮助下，HINet 在各种图像恢复任务上超越了最先进的 (SOTA)。对于图像去噪，我们在 SIDD 数据集上的 PSNR 中超过了 0.11dB 和 0.28dB，其乘法累加器操作 (MAC) 分别只有 7.5% 和 30%，加速分别为 6.8 倍和 2.9 倍。 F 或图像去模糊，我们在其 MAC 的 22.5% 和 REDS 和 GoPro 数据集上的 3.3 倍加速方面获得了相当的性能。 F 或图像去雨，我们在具有 1.4 倍加速比的多个数据集的平均结果上超过了 0.3 dB 的 PSNR。借助 HINet，我们赢得了 NTIRE 2021 图像去模糊挑战赛 - Track2 的第一名。 JPEG 伪影，PSNR 为 29.70。

Interactive Multi-Dimension Modulation with Dynamic Controllable Residual Learning for Image Restoration

论文名称：Interactive Multi-Dimension Modulation with Dynamic Controllable Residual Learning for Image Restoration

论文下载：https://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123650052.pdf

论文年份：ECCV 2020

论文被引：8（2022/05/04）

论文代码：https://github.com/hejingwenhejingwen/CResMD

Interactive image restoration aims to generate restored images by adjusting a controlling coefficient which determines the restoration level. Previous works are restricted in modulating image with a single coefficient. However, real images always contain multiple types of degradation, which cannot be well determined by one coefficient. To make a step forward, this paper presents a new problem setup, called multi-dimension (MD) modulation, which aims at modulating output effects across multiple degradation types and levels. Compared with the previous single-dimension (SD) modulation, the MD is setup to handle multiple degradations adaptively and relief unbalanced learning problem in different degradations. We also propose a deep architecture - CResMD with newly introduced controllable residual connections for multidimension modulation. Specifically, we add a controlling variable on the conventional residual connection to allow a weighted summation of input and residual. The values of these weights are generated by another condition network. We further propose a new data sampling strategy based on beta distribution to balance different degradation types and levels. With corrupted image and degradation information as inputs, the network can output the corresponding restored image. By tweaking the condition vector, users can control the output effects in MD space at test time. Extensive experiments demonstrate that the proposed CResMD achieve excellent performance on both SD and MD modulation tasks.

交互式图像恢复旨在通过调整确定恢复级别的控制系数来生成恢复的图像。以前的工作仅限于使用单个系数调制图像。然而，真实图像总是包含多种类型的退化，这些退化不能由一个系数很好地确定。为了向前迈出一步，本文提出了一种新的问题设置，称为多维 (MD) 调制，旨在跨多个退化类型和级别调制输出效果。与之前的单维 (SD) 调制相比，MD 被设置为自适应地处理多个退化并缓解不同退化中的不平衡学习问题。我们还提出了一种深度架构——CResMD，它具有新引入的用于多维调制的可控残差连接。具体来说，我们在传统的残差连接上添加了一个控制变量，以允许对输入和残差进行加权求和。这些权重的值由另一个条件网络生成。我们进一步提出了一种基于 beta 分布的新数据采样策略，以平衡不同的退化类型和水平。以损坏的图像和退化信息作为输入，网络可以输出相应的恢复图像。通过调整条件向量，用户可以在测试时控制 MD 空间中的输出效果。大量实验表明，所提出的 CResMD 在 SD 和 MD 调制任务上都取得了出色的性能。

Plug-and-Play Image Restoration with Deep Denoiser Prior

论文名称：Plug-and-Play Image Restoration with Deep Denoiser Prior

论文下载：https://ieeexplore.ieee.org/abstract/document/9454311

论文年份：TPAMI 2021

论文被引：93（2022/05/04）

论文代码：https://github.com/cszn/DPIR

Recent works on plug-and-play image restoration have shown that a denoiser can implicitly serve as the image prior for model-based methods to solve many inverse problems. Such a property induces considerable advantages for plug-and-play image restoration (e.g., integrating the flexibility of model-based method and effectiveness of learning-based methods) when the denoiser is discriminatively learned via deep convolutional neural network (CNN) with large modeling capacity. However, while deeper and larger CNN models are rapidly gaining popularity, existing plug-and-play image restoration hinders its performance due to the lack of suitable denoiser prior. In order to push the limits of plug-and-play image restoration, we set up a benchmark deep denoiser prior by training a highly flexible and effective CNN denoiser. We then plug the deep denoiser prior as a modular part into a half quadratic splitting based iterative algorithm to solve various image restoration problems. We, meanwhile, provide a thorough analysis of parameter setting, intermediate results and empirical convergence to better understand the working mechanism. Experimental results on three representative image restoration tasks, including deblurring, super-resolution and demosaicing, demonstrate that the proposed plug-and-play image restoration with deep denoiser prior not only significantly outperforms other state-of-the-art model-based methods but also achieves competitive or even superior performance against state-of-the-art learning-based methods.

最近关于即插即用图像恢复的工作表明，降噪器可以隐式地用作基于模型的方法的图像先验，以解决许多逆问题。当通过具有大型建模的深度卷积神经网络 (CNN) 有区别地学习去噪器时，这种特性为即插即用图像恢复带来了相当大的优势（例如，集成了基于模型的方法的灵活性和基于学习的方法的有效性）容量。然而，虽然更深、更大的 CNN 模型正在迅速普及，但现有的即插即用图像恢复由于缺乏合适的先验降噪器而阻碍了其性能。为了突破即插即用图像恢复的极限，我们通过训练高度灵活和有效的 CNN 去噪器来建立基准深度去噪器。然后，我们将深度去噪先验作为模块化部分插入基于半二次分裂的迭代算法中，以解决各种图像恢复问题。同时，我们对参数设置、中间结果和经验收敛进行了深入分析，以更好地理解工作机制。三个代表性图像恢复任务（包括去模糊、超分辨率和去马赛克）的实验结果表明，所提出的具有深度去噪先验的即插即用图像恢复不仅显着优于其他最先进的基于模型的方法，而且与最先进的基于学习的方法相比，它还实现了具有竞争力甚至优越的性能。

Path-Restore: Learning Network Path Selection for Image Restoration

论文名称：Path-Restore: Learning Network Path Selection for Image Restoration

论文下载：https://ieeexplore.ieee.org/document/9483659

论文年份：TPAMI 2021

论文被引：42（2022/05/04）

论文代码：https://github.com/yuke93/Path-Restore

Very deep Convolutional Neural Networks (CNNs) have greatly improved the performance on various image restoration tasks. However, this comes at a price of increasing computational burden, hence limiting their practical usages. We observe that some corrupted image regions are inherently easier to restore than others since the distortion and content vary within an image. T o leverage this, we propose Path-Restore, a multi-path CNN with a pathfinder that can dynamically select an appropriate route for each image region. We train the pathfinder using reinforcement learning with a difficulty-regulated reward. This reward is related to the performance, complexity and “the difficulty of restoring a region”. A policy mask is further investigated to jointly process all the image regions. We conduct experiments on denoising and mixed restoration tasks. The results show that our method achieves comparable or superior performance to existing approaches with less computational cost. In particular, Path-Restore is effective for real-world denoising, where the noise distribution varies across different regions on a single image. Compared to the state-of-the-art RIDNet [1], our method achieves comparable performance and runs 2.7x faster on the realistic Darmstadt Noise Dataset [2]. Models and codes will be released.

非常深的卷积神经网络 (CNN) 极大地提高了各种图像恢复任务的性能。然而，这是以增加计算负担为代价的，因此限制了它们的实际用途。我们观察到一些损坏的图像区域本质上比其他区域更容易恢复，因为图像中的失真和内容会有所不同。为了利用这一点，我们提出了 Path-Restore，这是一种多路径 CNN，带有一个路径查找器，可以为每个图像区域动态选择合适的路径。我们使用具有难度调节奖励的强化学习来训练探路者。该奖励与性能、复杂性和“恢复区域的难度”有关。进一步研究策略掩码以联合处理所有图像区域。我们对去噪和混合恢复任务进行了实验。结果表明，我们的方法以更少的计算成本实现了与现有方法相当或更高的性能。特别是，路径还原对于真实世界的去噪非常有效，其中噪声分布在单个图像的不同区域之间变化。与最先进的 RIDNet [1] 相比，我们的方法实现了相当的性能，并且在真实的达姆施塔特噪声数据集 [2] 上运行速度提高了 2.7 倍。模型和代码将被发布。

Residual Dense Network for Image Restoration

论文名称：Residual Dense Network for Image Restoration

论文下载：https://ieeexplore.ieee.org/abstract/document/8964437

论文年份：TPAMI 2018

论文被引：272（2022/05/04）

论文代码：https://github.com/yulunzhang/RDN

Recently, deep convolutional neural network (CNN) has achieved great success for image restoration (IR) and provided hierarchical features at the same time. However, most deep CNN based IR models do not make full use of the hierarchical features from the original low-quality images; thereby, resulting in relatively-low performance. In this work, we propose a novel and efficient residual dense network (RDN) to address this problem in IR, by making a better tradeoff between efficiency and effectiveness in exploiting the hierarchical features from all the convolutional layers. Specifically, we propose residual dense block (RDB) to extract abundant local features via densely connected convolutional layers. RDB further allows direct connections from the state of preceding RDB to all the layers of current RDB, leading to a contiguous memory mechanism. To adaptively learn more effective features from preceding and current local features and stabilize the training of wider network, we proposed local feature fusion in RDB. After fully obtaining dense local features, we use global feature fusion to jointly and adaptively learn global hierarchical features in a holistic way. We demonstrate the effectiveness of RDN with several representative IR applications, single image super-resolution, Gaussian image denoising, image compression artifact reduction, and image deblurring. Experiments on benchmark and real-world datasets show that our RDN achieves favorable performance against state-of-the-art methods for each IR task quantitatively and visually.

最近，深度卷积神经网络 (CNN) 在图像恢复 (IR) 方面取得了巨大成功，同时提供了分层特征。然而，大多数基于深度 CNN 的 IR 模型并没有充分利用原始低质量图像的层次特征；因此，导致性能相对较低。在这项工作中，我们提出了一种新颖且高效的残差密集网络（RDN）来解决 IR 中的这个问题，通过在利用所有卷积层的分层特征时在效率和有效性之间做出更好的权衡。具体来说，我们提出残差密集块（RDB）通过密集连接的卷积层提取丰富的局部特征。 RDB 进一步允许从前一个 RDB 的状态直接连接到当前 RDB 的所有层，从而形成一个连续的内存机制。为了自适应地从先前和当前的局部特征中学习更有效的特征并稳定更广泛网络的训练，我们提出了 RDB 中的局部特征融合。在充分获得密集的局部特征后，我们使用全局特征融合，以整体的方式联合和自适应地学习全局层次特征。我们通过几个具有代表性的 IR 应用、单图像超分辨率、高斯图像去噪、图像压缩伪影减少和图像去模糊证明了 RDN 的有效性。在基准和真实世界数据集上的实验表明，我们的 RDN 在定量和视觉上针对每个 IR 任务的最先进方法实现了良好的性能。

Dual Residual Networks Leveraging the Potential of Paired Operations for Image Restoration

论文名称：Dual Residual Networks Leveraging the Potential of Paired Operations for Image Restoration

论文下载：link

论文年份：CVPR 2019

论文被引：100（2022/05/04）

论文代码：https://github.com/liu-vis/DualResidualNetworks

In this paper , we study design of deep neural networks for tasks of image restoration. We propose a novel style of residual connections dubbed “dual residual connection”, which exploits the potential of paired operations, e.g., upand down-sampling or convolution with large- and smallsize kernels. We design a modular block implementing this connection style; it is equipped with two containers to which arbitrary paired operations are inserted. Adopting the “unraveled” view of the residual networks proposed by V eit et al., we point out that a stack of the proposed modular blocks allows the first operation in a block interact with the second operation in any subsequent blocks. Specifying the two operations in each of the stacked blocks, we build a complete network for each individual task of image restoration. We experimentally evaluate the proposed approach on five image restoration tasks using nine datasets. The results show that the proposed networks with properly chosen paired operations outperform previous methods on almost all of the tasks and datasets.

在本文中，我们研究了用于图像恢复任务的深度神经网络的设计。我们提出了一种新颖的残差连接样式，称为“双残差连接”，它利用了配对操作的潜力，例如，上采样和下采样或使用大小内核进行卷积。我们设计了一个实现这种连接方式的模块化块；它配备了两个可插入任意配对操作的容器。采用 Veit 等人提出的残差网络的“未分解”视图，我们指出，所提出的模块化块的堆栈允许块中的第一个操作与任何后续块中的第二个操作交互。在每个堆叠块中指定两个操作，我们为每个图像恢复任务构建一个完整的网络。我们使用九个数据集在五个图像恢复任务上对所提出的方法进行了实验评估。结果表明，在几乎所有任务和数据集上，具有正确选择配对操作的建议网络都优于以前的方法。

Exploiting the Potential of Standard Convolutional Autoencoders for Image Restoration by Evolutionary Search

论文名称：Exploiting the Potential of Standard Convolutional Autoencoders for Image Restoration by Evolutionary Search

论文下载：http://proceedings.mlr.press/v80/suganuma18a.html

论文年份：PMLR 2018

论文被引：70（2022/05/04）

论文代码：https://github.com/sg-nm/Evolutionary-Autoencoders

Researchers have applied deep neural networks to image restoration tasks, in which they proposed various network architectures, loss functions, and training methods. In particular, adversarial training, which is employed in recent studies, seems to be a key ingredient to success. In this paper, we show that simple convolutional autoencoders (CAEs) built upon only standard network components, i.e., convolutional layers and skip connections, can outperform the state-of-the-art methods which employ adversarial training and sophisticated loss functions. The secret is to search for good architectures using an evolutionary algorithm. All we did was to train the optimized CAEs by minimizing the ‘2 loss between reconstructed images and their ground truths using the ADAM optimizer. Our experimental results show that this approach achieves 27.8 dB peak signal to noise ratio (PSNR) on the CelebA dataset and 33.3 dB on the SVHN dataset, compared to 22.8 dB and 19.0 dB provided by the former state-of-the-art methods, respectively.

研究人员将深度神经网络应用于图像恢复任务，提出了各种网络架构、损失函数和训练方法。特别是，最近研究中采用的对抗性训练似乎是成功的关键因素。在本文中，我们展示了仅基于标准网络组件（即卷积层和跳过连接）构建的简单卷积自动编码器（CAE）可以优于采用对抗训练和复杂损失函数的最先进方法。秘诀是使用进化算法搜索好的架构。我们所做的只是通过使用 ADAM 优化器最小化重建图像与其基本事实之间的“2”损失来训练优化的 CAE。我们的实验结果表明，这种方法在 CelebA 数据集上实现了 27.8 dB 的峰值信噪比 (PSNR)，在 SVHN 数据集上实现了 33.3 dB，而之前的最先进方法分别为 22.8 dB 和 19.0 dB。

Crafting a Toolchain for Image Restoration by Deep Reinforcement Learning

论文名称：Crafting a Toolchain for Image Restoration by Deep Reinforcement Learning

论文下载：link

论文年份：CVPR 2018

论文被引：130（2022/05/04）

论文代码：https://github.com/yuke93/RL-Restore

We investigate a novel approach for image restoration by reinforcement learning. Unlike existing studies that mostly train a single large network for a specialized task, we prepare a toolbox consisting of small-scale convolutional networks of different complexities and specialized in different tasks. Our method, RL-Restore, then learns a policy to select appropriate tools from the toolbox to progressively restore the quality of a corrupted image. We formulate a stepwise reward function proportional to how well the image is restored at each step to learn the action policy. We also devise a joint learning scheme to train the agent and tools for better performance in handling uncertainty. In comparison to conventional human-designed networks, RL-Restore is capable of restoring images corrupted with complex and unknown distortions in a more parameter-efficient manner using the dynamically formed toolchain.

我们研究了一种通过强化学习进行图像恢复的新方法。与现有研究主要针对特定任务训练单个大型网络不同，我们准备了一个工具箱，该工具箱由具有不同复杂性并专门用于不同任务的小规模卷积网络组成。我们的方法，RL-Restore，然后学习一种策略，从工具箱中选择适当的工具，以逐步恢复损坏图像的质量。我们制定了一个逐步奖励函数，该函数与每个步骤中图像恢复的好坏成正比，以学习动作策略。我们还设计了一个联合学习方案来训练代理和工具，以更好地处理不确定性。与传统的人工设计网络相比，RL-Restore 能够使用动态形成的工具链以更有效的参数方式恢复因复杂和未知失真而损坏的图像。

Multi-Level Wavelet-CNN for Image Restoration

论文名称：Multi-Level Wavelet-CNN for Image Restoration

论文下载：link

论文年份：CVPR 2018

论文被引：371（2022/05/04）

论文代码：https://github.com/lpj0/MWCNN

The tradeoff between receptive field size and efficiency is a crucial issue in low level vision. Plain convolutional networks (CNNs) generally enlarge the receptive field at the expense of computational cost. Recently, dilated filtering has been adopted to address this issue. But it suffers from gridding effect, and the resulting receptive field is only a sparse sampling of input image with checkerboard patterns. In this paper, we present a novel multi-level wavelet CNN (MWCNN) model for better tradeoff between receptive field size and computational efficiency. With the modified U-Net architecture, wavelet transform is introduced to reduce the size of feature maps in the contracting subnetwork. Furthermore, another convolutional layer is further used to decrease the channels of feature maps. In the expanding subnetwork, inverse wavelet transform is then deployed to reconstruct the high resolution feature maps. Our MWCNN can also be explained as the generalization of dilated filtering and subsampling, and can be applied to many image restoration tasks. The experimental results clearly show the effectiveness of MWCNN for image denoising, single image super-resolution, and JPEG image artifacts removal.

感受野大小和效率之间的权衡是低水平视觉中的一个关键问题。普通卷积网络 (CNN) 通常以牺牲计算成本为代价扩大感受野。最近，已经采用膨胀滤波来解决这个问题。但它会受到网格效应的影响，由此产生的感受野只是具有棋盘格图案的输入图像的稀疏采样。在本文中，我们提出了一种新颖的 multi-level wavelet CNN（MWCNN）模型，以更好地平衡感受野大小和计算效率。通过修改后的 U-Net 架构，引入了小波变换以减小收缩子网络中特征图的大小。此外，另一个卷积层进一步用于减少特征图的通道。在扩展的子网络中，然后部署逆小波变换来重建高分辨率特征图。我们的 MWCNN 也可以解释为扩张滤波和二次采样的泛化，并且可以应用于许多图像恢复任务。实验结果清楚地表明了 MWCNN 在图像去噪、单图像超分辨率和 JPEG 图像伪影去除方面的有效性。

Non-Local Recurrent Network for Image Restoration

论文名称：Non-Local Recurrent Network for Image Restoration

论文下载：https://arxiv.org/abs/1806.02919

论文年份：NIPS 2018

论文被引：378（2022/05/04）

论文代码：https://github.com/Ding-Liu/NLRN

Many classic methods have shown non-local self-similarity in natural images to be an effective prior for image restoration. However, it remains unclear and challenging to make use of this intrinsic property via deep networks. In this paper, we propose a non-local recurrent network (NLRN) as the first attempt to incorporate non-local operations into a recurrent neural network (RNN) for image restoration. The main contributions of this work are: (1) Unlike existing methods that measure self-similarity in an isolated manner, the proposed non-local module can be flexibly integrated into existing deep networks for end-to-end training to capture deep feature correlation between each location and its neighborhood. (2) We fully employ the RNN structure for its parameter efficiency and allow deep feature correlation to be propagated along adjacent recurrent states. This new design boosts robustness against inaccurate correlation estimation due to severely degraded images. (3) We show that it is essential to maintain a confined neighborhood for computing deep feature correlation given degraded images. This is in contrast to existing practice [41] that deploys the whole image. Extensive experiments on both image denoising and super-resolution tasks are conducted. Thanks to the recurrent non-local operations and correlation propagation, the proposed NLRN achieves superior results to state-of-the-art methods with many fewer parameters.

许多经典方法已经表明自然图像中的非局部自相似性是图像恢复的有效先验。然而，通过深度网络利用这种内在属性仍然不清楚且具有挑战性。在本文中，我们提出了一种非局部循环网络（NLRN）作为将非局部操作合并到循环神经网络（RNN）中进行图像恢复的第一次尝试。这项工作的主要贡献是：（1）与现有的以孤立方式测量自相似性的方法不同，所提出的非局部模块可以灵活地集成到现有的深度网络中进行端到端训练，以捕获深度特征相关性在每个位置及其附近。 (2) 我们充分利用 RNN 结构的参数效率，并允许沿相邻循环状态传播深度特征相关性。这种新设计提高了对由于图像严重退化而导致的不准确相关估计的鲁棒性。 (3) 我们表明，在给定退化图像的情况下，必须保持一个受限邻域来计算深度特征相关性。这与部署整个图像的现有实践 [41] 形成对比。对图像去噪和超分辨率任务进行了广泛的实验。由于经常性的非局部操作和相关传播，所提出的 NLRN 以更少的参数实现了优于最先进方法的结果。

Noise2Noise: Learning Image Restoration without Clean Data

论文名称：Noise2Noise: Learning Image Restoration without Clean Data

论文下载：https://arxiv.org/abs/1803.04189

论文年份：PMLR 2018

论文被引：767（2022/05/04）

论文代码：https://github.com/NVlabs/noise2noise

We apply basic statistical reasoning to signal reconstruction by machine learning – learning to map corrupted observations to clean signals – with a simple and powerful conclusion: it is possible to learn to restore images by only looking at corrupted examples, at performance at and sometimes exceeding training using clean data, without explicit image priors or likelihood models of the corruption. In practice, we show that a single model learns photographic noise removal, denoising synthetic Monte Carlo images, and reconstruction of undersampled MRI scans – all corrupted by different processes – based on noisy data only.

我们通过机器学习将基本的统计推理应用于信号重建——学习将损坏的观察映射到干净的信号——得出一个简单而有力的结论：可以通过仅查看损坏的示例来学习恢复图像，性能达到甚至有时超过训练使用干净的数据，没有明确的图像先验或损坏的可能性模型。在实践中，我们展示了单个模型仅基于噪声数据学习摄影噪声去除、合成蒙特卡罗图像去噪和欠采样 MRI 扫描的重建——所有这些都被不同的过程破坏。

COLA-Net: Collaborative Attention Network for Image Restoration

论文名称：COLA-Net: Collaborative Attention Network for Image Restoration

论文下载：https://arxiv.org/abs/2103.05961

论文年份：IEEE Transactions on Multimedia 2021

论文被引：12（2022/05/04）

论文代码：https://github.com/MC-E/COLA-Net-Collaborative-Attention-Network-for-Image-Restoration

Local and non-local attention-based methods have been well studied in various image restoration tasks while leading to promising performance. However, most of the existing methods solely focus on one type of attention mechanism (local or nonlocal). Furthermore, by exploiting the self-similarity of natural images, existing pixel-wise non-local attention operations tend to give rise to deviations in the process of characterizing longrange dependence due to image degeneration. To overcome these problems, in this paper we propose a novel collaborative attention network (COLA-Net) for image restoration, as the first attempt to combine local and non-local attention mechanisms to restore image content in the areas with complex textures and with highly repetitive details respectively. In addition, an effective and robust patch-wise non-local attention model is developed to capture longrange feature correspondences through 3D patches. Extensive experiments on synthetic image denoising, real image denoising and compression artifact reduction tasks demonstrate that our proposed COLA-Net is able to achieve state-of-the-art performance in both peak signal-to-noise ratio and visual perception, while maintaining an attractive computational complexity.

基于局部和非局部注意的方法已经在各种图像恢复任务中得到了很好的研究，同时带来了有希望的性能。然而，大多数现有方法只关注一种类型的注意力机制（局部或非局部）。此外，通过利用自然图像的自相似性，现有的逐像素非局部注意力操作往往会在表征长距离依赖性的过程中由于图像退化而产生偏差。为了克服这些问题，在本文中，我们提出了一种用于图像恢复的新型协同注意力网络（COLA-Net），作为首次尝试结合局部和非局部注意力机制来分别恢复具有复杂纹理和高度的区域的图像内容。此外，还开发了一种有效且稳健的补丁式非局部注意模型，以通过 3D 补丁捕获远程特征对应关系。对合成图像去噪、真实图像去噪和压缩伪影减少任务的大量实验表明，我们提出的 COLA-Net 能够在峰值信噪比和视觉感知方面实现最先进的性能，同时保持有吸引力的计算复杂性。

Hyperspectral Image Restoration by Tensor Fibered Rank Constrained Optimization and Plug-and-Play Regularization

论文名称：Hyperspectral Image Restoration by Tensor Fibered Rank Constrained Optimization and Plug-and-Play Regularization

论文下载：https://ieeexplore.ieee.org/abstract/document/9314228

论文年份：TGRS 2022

论文被引：23（2022/05/04）

论文代码：——

Hyperspectral images (HSIs) are often contaminated by several types of noise, which significantly limits the accuracy of subsequent applications. Recently, low-rank modeling based on tensor singular value decomposition (T-SVD) has achieved great success in HSI restoration. Most of them use the convex and nonconvex surrogates of the tensor rank, which cannot well approximate the tensor singular values and obtain suboptimal restored results. We suggest a novel HSI restoration model by introducing a fibered rank constrained tensor restoration framework with an embedded plug-and-play (PnP)-based regularization (FRCTR-PnP). More precisely, instead of using the convex and nonconvex surrogates to approximate the fibered rank, the proposed model directly constrains the tensor fibered rank of the solution, leading to a better approximation to the original image. Since exploiting the low-fibered-rankness of HSI is mainly to capture the global structure, we further employ an implicit PnP-based regularization to preserve the image details. Particularly, the above two building blocks are complementary to each other, rather than isolated and uncorrelated. Based on the alternating direction multiplier method (ADMM), we propose an efficient algorithm to tackle the proposed model. For robustness, we develop a three-directional randomized T-SVD (3DRT-SVD), which preserves the intrinsic structure of the clean HSI and removes partial noise by projecting the HSI onto a lowdimensional essential subspace. Extensive experimental results including simulated and real data demonstrate that the proposed method achieves superior performance over compared methods in terms of quantitative evaluation and visual inspection.

高光谱图像（HSI）经常被几种类型的噪声污染，这极大地限制了后续应用的准确性。最近，基于张量奇异值分解（T-SVD）的低秩建模在 HSI 恢复方面取得了巨大成功。他们中的大多数使用张量秩的凸和非凸代理，不能很好地逼近张量奇异值并获得次优的恢复结果。我们通过引入基于嵌入式即插即用 (PnP) 正则化 (FRCTR-PnP) 的纤维秩约束张量恢复框架来提出一种新的 HSI 恢复模型。更准确地说，所提出的模型不是使用凸和非凸代理来逼近纤维秩，而是直接约束解的张量纤维秩，从而更好地逼近原始图像。由于利用 HSI 的低纤维秩主要是为了捕获全局结构，我们进一步采用基于 PnP 的隐式正则化来保留图像细节。特别是，上述两个构建块是相互补充的，而不是孤立的和不相关的。基于交替方向乘法器方法（ADMM），我们提出了一种有效的算法来解决所提出的模型。为了鲁棒性，我们开发了一种三向随机 T-SVD (3DRT-SVD)，它保留了干净 HSI 的内在结构，并通过将 HSI 投影到低维基本子空间来消除部分噪声。包括模拟和真实数据在内的大量实验结果表明，所提出的方法在定量评估和视觉检查方面比比较方法具有更好的性能。

HSI-DeNet: Hyperspectral Image Restoration via Convolutional Neural Network

论文名称：HSI-DeNet: Hyperspectral Image Restoration via Convolutional Neural Network

论文下载：https://ieeexplore.ieee.org/abstract/document/8435923

论文年份：TGRS 2018

论文被引：113（2022/05/04）

论文代码：https://github.com/junjun-jiang/Hyperspectral-Image-Denoising-Benchmark

The spectral and the spatial information in hyperspectral images (HSIs) are the two sides of the same coin. How to jointly model them is the key issue for HSIs’ noise removal, including random noise, structural stripe noise, and dead pixels/lines. In this paper , we introduce the deep convolutional neural network (CNN) to achieve this goal. The learned filters can well extract the spatial information within their local receptive filed. Meanwhile, the spectral correlation can be depicted by the multiple channels of the learned 2-D filters, namely, the number of filters in each layer. The consequent advantages of our CNN-based HSI denoising method (HSI-DeNet) over previous methods are threefold. First, the proposed HSI-DeNet can be regarded as a tensor-based method by directly learning the filters in each layer without damaging the spectral-spatial structures. Second, the HSI-DeNet can simultaneously accommodate various kinds of noise in HSIs. Moreover, our method is flexible for both single image and multiple images by slightly modifying the channels of the filters in the first and last layers. Last but not least, our method is extremely fast in the testing phase, which makes it more practical for real application. The proposed HSI-DeNet is extensively evaluated on several HSIs, and outperforms the state-of-the-art HSI-DeNets in terms of both speed and performance.

高光谱图像 (HSI) 中的光谱和空间信息是同一枚硬币的两个面。如何对它们进行联合建模是 HSI 去噪的关键问题，包括随机噪声、结构条纹噪声和坏点/线。在本文中，我们介绍了深度卷积神经网络 (CNN) 来实现这一目标。学习到的过滤器可以很好地提取其局部感受野内的空间信息。同时，光谱相关性可以用学习到的二维滤波器的多个通道来描述，即每层滤波器的数量。我们基于 CNN 的 HSI 去噪方法 (HSI-DeNet) 与以前的方法相比具有三倍的优势。首先，所提出的 HSI-DeNet 可以被视为一种基于张量的方法，通过直接学习每一层中的滤波器而不破坏光谱空间结构。其次，HSI-DeNet 可以同时适应 HSI 中的各种噪声。此外，通过稍微修改第一层和最后一层中滤波器的通道，我们的方法对单张图像和多张图像都很灵活。最后但同样重要的是，我们的方法在测试阶段非常快，这使得它在实际应用中更加实用。所提出的 HSI-DeNet 在几个 HSI 上进行了广泛评估，并且在速度和性能方面都优于最先进的 HSI-DeNet。

EDPN: Enhanced Deep Pyramid Network for Blurry Image Restoration

论文名称：EDPN: Enhanced Deep Pyramid Network for Blurry Image Restoration

论文下载：https://arxiv.org/abs/2105.04872

论文年份：CVPRW 2021

论文被引：5（2022/05/04）

论文代码：https://github.com/zeyuxiao1997/EDPN

Image Restoration Using Total Variation Regularized Deep Image Prior

论文名称：Image Restoration Using Total Variation Regularized Deep Image Prior

论文下载：https://ieeexplore.ieee.org/abstract/document/8682856

论文年份：ICASSP 2019

论文被引：79（2022/05/04）

论文代码：——

Image Restoration by Iterative Denoising and Backward Projections

论文名称：Image Restoration by Iterative Denoising and Backward Projections

论文下载：https://ieeexplore.ieee.org/abstract/document/8489894

论文年份：TIP 2018

论文被引：110（2022/05/04）

论文代码：https://github.com/tomtirer/IDBP （matlab）

Multi-scale adversarial network for underwater image restoration

论文名称：Multi-scale adversarial network for underwater image restoration

论文下载：https://www.sciencedirect.com/journal/optics-and-laser-technology

论文年份：2019

论文被引：（2022/05/04）

论文代码：

Deep learning–based image restoration algorithm for coronary CT angiography

论文名称：Deep learning–based image restoration algorithm for coronary CT angiography

论文下载：https://link.springer.com/article/10.1007/s00330-019-06183-y

论文年份：2019

论文被引：（2022/05/04）

论文代码：

论文名称：

论文下载：

论文年份：20

论文被引：（2022/05/04）

论文代码：

Others

Title	Year	Code	Abstract	Source
Deep Learning with Inaccurate Training Data for Image Restoration	2018			https://zhuanlan.zhihu.com/p/144874715
Variational AutoEncoder for Reference based Image Super-Resolution	2021	https://github.com/Holmes-Alan/RefVAE		https://zhuanlan.zhihu.com/p/381076261
Robust Reference-based Super-Resolution via -Matching	2021			https://zhuanlan.zhihu.com/p/380326467
Fourier Space Losses for Efficient Perceptual Image Super-Resolution	2021			https://zhuanlan.zhihu.com/p/380238138
Content-adaptive Representation Learning for Fast Image Super-resolution	2021			https://zhuanlan.zhihu.com/p/374395519
Joint Face Image Restoration and Frontalization for Recognition	2021			https://zhuanlan.zhihu.com/p/374248027
HINet: Half Instance Normalization Network for Image Restoration	2021			https://zhuanlan.zhihu.com/p/372341817
EDPN: Enhanced Deep Pyramid Network for Blurry Image Restoration	2021			https://zhuanlan.zhihu.com/p/371802390
Toward Interactive Modulation for Photo-Realistic Image Restoration	2021			https://zhuanlan.zhihu.com/p/371520342
Simultaneous Face Hallucination and Translation for Thermal to Visible Face Verification using Axial-GAN	2021	https://github.com/sam575/axial-gan		https://zhuanlan.zhihu.com/p/365255454
COLA-Net: Collaborative Attention Network for Image Restoration	2021			https://zhuanlan.zhihu.com/p/362206420
Learning Frequency-aware Dynamic Network for Efficient Super-Resolution	2021			https://zhuanlan.zhihu.com/p/361110281
Best-Buddy GANs for Highly Detailed Image Super-Resolution	2021			https://zhuanlan.zhihu.com/p/361409850
BaMBNet: A Blur-aware Multi-branch Network for Defocus Deblurring	2021			https://zhuanlan.zhihu.com/p/377441340
GAN Prior Embedded Network for Blind Face Restoration in the Wild	2021			https://zhuanlan.zhihu.com/p/372348930
High-Resolution Photorealistic Image Translation in Real-Time: A Laplacian Pyramid Translation Network	2021			https://zhuanlan.zhihu.com/p/373971032
Identity and Attribute Preserving Thumbnail Upscaling	2021			https://zhuanlan.zhihu.com/p/379704404
SDNet: mutil-branch for single image deraining using swin	2021			https://zhuanlan.zhihu.com/p/379687950