Faster Read: Learning a Deep Single Image Contrast Enhancer from Multi-Exposure Images
Single Image Contrast Enhancer IEEE Transactions on Image Processing April 2018
keywords: CNN(Convolutional Neural Network), SICE，MEF
论文名称：
Learning a Deep Single Image Contrast Enhancer from Multi-Exposure Images
Prerequisite：Convolutional Neural Network
提高图像对比度的方法分单图像提高对比度算法(SICE)和同一场景多个曝光度合成的算法(MEF)。MEF之后的图片可以展现更多的图片细节，但是由于要拍摄多个曝光度的原因，如果拍摄中有动态的物体的话，合成后会出现虚影(ghosting)。另一方面，SICE可以只用一张图强化图片对比度，所以不会出现ghosting。但是合成的效果不如MEF，图片细节也不如MEF出来的图片效果好。有没有一种方法，既能没有ghosting，又能合成高对比度的图片呢？机器学习(ML)正好能做到这一点。如果输入是一张图片，label是同一场景下MEF合成的图片，通过适当的学习就能达到MEF的效果。这篇文章有灵性的地方就在于巧妙的运用ML架起了桥，绕过了这个坎。
目的：用新的提取图像特征的方法，结合不同的loss function用做实时的图像风格转化和超分辨率
重点和主要贡献：
- 提出了一个多曝光度的hdr图片的数据库，叫SICE。
- 用learning-based算法实现SICE，而且能达到MEF的效果。
==================================================================================
下面是部分重点的细节介绍，上面看懂的不用再往下看了。
==================================================================================
Review：
SICE方法：Histogram-based algorithms [4]–[6], which increase the contrast of an image by redistributing the luminous intensity on histogram. Retinex based algorithms [7]–[9], which enhance the reflectance and illumination components of the image separately.
以上SICE的缺点：
- 复杂的自然场景和有限的图像信息很难恢复：These methods, however, are difficult to reproduce a high-quality image due to the complex natural scenes and the limited information in a single low contrast image.
- 重新分配发光强度会使图片强烈失真：Histogram-based methods [4], [5] attempt to redistribute the luminous intensity on histogram in a global or local manner. However, such simple redistribution operations may produce serious unrealistic effects in the enhanced images since they ignore image structural information [26]
- 多数SICE算法是基于图片是高质量图片的假设，而没有充分利用(exploit)图像信息(这是作者说的)：Most of the previous SICE methods are based on some assumptions on high-quality images, while they may not fully exploit the information in the input image.
- [9]说SICE增强能力有限，因为受限于低对比度图像的有限的信息：the enhancement capability of existing SICE methods is rather limited due to the limited information in a single low-contrast image .
MEF方法: [10], [11], development of imaging devices are able to capture a sequence of multi-exposure images in a short time to fulfil the dynamic range of a scene. Multi-exposure image fusion (MEF) [2], [12], [13] and stack-based high dynamic range (HDR) imaging methods [14], [15] can be applied to blend the multiple images with different exposures into a perceptually more appealing image.
MEF和stack-based HDR方法的区别：stack-based HDR merges bracketed multiple exposure images into an HDR irradiance map, then employ a tone mapping operator to compress the dynamic range of HDR irradiance map so that the high-contrast image can be displayed on regular monitors. MEF methods attempt to fuse the images directly in the non-liner brightness domain to reproduce a high-visibility image.
电子图像因为动态范围而失去细节：[30] Because of the limited dynamic range, traditional digital imaging systems may lose structural details when shooting a natural scene.
CNN方法：Methods [28], [31] have been proposed to train a CNN network to map the low dynamic range (LDR) images to HDR images. In [29], a CNN is trained to set the parameters of bilateral filters, which are then used to enhance an input image to a desired image edited by professional photographers.
Ghosting问题: [16], [17] the acquisition of multi-exposure images will complicate the imaging process, and camera shake or moving objects will lead to unpleasant fusion artifacts such as the ghosting artifacts .
Deghosting方法: [2], [14], [32],其中[2] is the state-of-the-art deghosting MEF algorithm. Learning-based methods like [33] proposed to map the multi-exposure image sequences to an HDR image.
SICE是为了提高DR下拍摄的低对比度图像的对比度：[25] Single image contrast enhancement (SICE) aims to improve the visibility of the scene in a given single low-contrast image. It provides a way to enhance the low contrast photographs captured from a high dynamic range scene。
CNN的用途：
- 去噪 denoising [34], [35]
- 超分辨率 super-resolution [21]
- 去模糊 deblurring [36]
以上应用很容易弄出训练数据和高质量的label：In those applications, pairs of degraded images and their high-quality coun- terparts can be easily generated. With those paired training data, CNN can be used to learn a mapping function between the degraded observations and their corresponding high-quality reference images.
但是SICE这种基于规则的数据对于真实世界的低对比度图像太理想，CNN不好学习：However, for the application of SICE, computer-generated training datasets are too ideal to be true for real-world low-contrast images, where the distribution of luminance is much more complex and varies with different scenes, cameras and camera settings.
[37], [38]有多曝光度的数据库，但是数量少，不狗多样化：Some multi-exposure image sequences are available in literature [37], [38]. However, the total amount of such publicly available sequences is very limited, and many of them were taken under indoor environ- ment. Neither the number of sequences nor the diversity of sequence exposure levels meets the requirement of real-world applications.
收集数据的设备：Seven types of consumer grade cameras are used to collect the image sequences, including Sony α7RII, Sony NEX-5N, Canon EOS-5D Mark II, Canon EOS-750D, Nikon D810, Nikon D7100 and iPhone 6s.
跟13种MEF对比：13 state-of-the-art MEF and HDR algorithms are employed in this process, including 8 MEF methods: Mertens09 [13], Raman09 [39], Shen11 [40], Zhang12 [41], Li13 [12], Shen14 [42], Ma17 [2], Kou17 [43], and 5 stack- based HDR methods: Sen12 [14], Hu13 [32], Bruce14 [44], Oh15 [15], Photomatix [45].
用MEF-SSIM [16]对比评分，选出最高的做label。
作者尝试直接用DNN学习，像下面的网络结构，但是效果不佳。
[49]说低频信息代表整体自然度，高频信息代表局部细节：Retinex theory [49], the low-frequency information of an image represents the global naturalness, and the high-frequency information represents the local details.[50]–[53]拆解图像到高低频率：have been proposed to decompose an image into high and low frequency components to preserve image details and colors in the brightest/darkest regions.
然后作者有了想法，把图像拆成两个频率，低频用cnn加一次residual，高频用unet。分别算不同的loss。最后加在一起用第三个网络微调。然后作者觉得棒棒哒了~
网络细节如下：
作者用PSNR和FSIM [60]来衡量图片质量。用实验作对比。结构好像是依赖主观的想象，并没有做出深入的分析为什么用某个结构，某个loss，为什么用residual。
最后有意思的是作者列出了failure case(很严肃，绝对不是幸灾乐祸啊).
个人感觉，深入研究这些failure cases，才是发NB论文的正道。
作者对问题的描述：However, it is also found that our method may fail to recover the details for large and severely over- exposed regions. Figure 18 shows an example. One can see that the missing color and structures in the color chart are not recovered, while the details can be seen in the reference image generated by MEF/HDR methods.
作者认为是过曝光太严重了，区域又大，信息太少卷积不出来：The reason for the failure may be that the over-exposure is too severe (in terms of both level and area) so that there is little information the CNN can use to synthesize the missing details in the neighborhood.
Ref:
论文：https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8259342
发表于 Published in: IEEE Transactions on Image Processing April 2018