【论文笔记】LIME: low-light image enhancement via illumination map estimation
【论文笔记】LIME: low-light image enhancement via illumination map estimation
-
- Traditional Methods & Unsolved Problems
- Solutions
-
- Retinex
- Illumination Estimation
- Exploit Structure of illumination to REFINE
- Sped-Up Solver to Problem
- Possible Weighting Strategies
-
- simply set to 1
- set the entire coefficient to 1
- set by Relative Total Variation
- Other refine operations
-
- Gamma correction
- Denoising
- Highlights
Traditional Methods & Unsolved Problems
- Histogram equalization (HE)
- focus on contrast enhancement instead of exploiting real illumination causes, having the risk of over- and under- enhancement
- Gamma correction
- carried out on each pixel individually
- single-scale Retinex (SSR) and multi-scale Retinex (MSR)
- looks unnatural and frequently appears to be over-enhanced
- adjust the illumination by fusing multiple derivations of the initially estimated illumination map
- lose the realism of regions with rich textures
- dehaze inverted low-light image
- lacking in physical explanation
Solutions
- Retinex-based methods but only estimates one factor, say the illumination
- shrink the solution space and reduce the computational cost
- exploit the structure of illumination to refine illumination map
Retinex
L = R ⊙ T \mathbf {L=R \odot T} L=R⊙T
where
- L \mathbf L L are the captured low-light image
- R \mathbf R R are the desired recovery (aka reflectance)
- T \mathbf T T represents the illumination map
- ⊙ \odot ⊙ means element-wise multiplication
Assume: for color images, three channels share the same illumination map. T \mathbf T T and T ^ \hat {\mathbf T} T^ to represent one-channel and three-channel illumination maps.
Illumination Estimation
T ^ ( x ) ← max c ∈ { R , G , B } L c ( x ) (1) \hat {\mathbf T}(x) \leftarrow \max_{c∈\{R,G,B\}} \mathbf L ^c(x)\tag 1 T^(x)←c∈{R,G,B}maxLc(x)(1)
so R ( x ) = L ( x ) max c L c ( x ) + ϵ (2) \mathbf R(x)={\mathbf L(x)\over \max_c \mathbf L^c(x)+\epsilon}\tag 2 R(x)=maxcLc(x)+ϵL(x)(2)
In dehaze model:
1 − L = ( 1 − R ) ⊙ T ~ + a ( 1 − T ~ ) (3) \mathbf{1-L=(1-R)\odot\tilde T}+ a \mathbf{(1-\tilde T)} \tag 3 1−L=(1−R)⊙T~+a(1−T~)(3)
where
- 1 − L \mathbf{1-L} 1−L is inverted low-light image (similar to haze image)
with dark channel prior
T ~ ( x ) ← 1 − min c 1 − L c ( x ) a = 1 − 1 a + max c L c ( x ) a (4) \mathbf {\tilde T}(x) \leftarrow 1-\min_c{1-\mathbf L^c(x)\over a}=1-{1 \over a}+\max_c{\mathbf L^c(x)\over a}\tag 4 T~(x)←1−cmina1−Lc(x)=1−a1+cmaxaLc(x)(4)
substituting (2) into (1)
R ( x ) = L ( x ) − 1 + a ( 1 − 1 a + max c L c ( x ) a + ϵ ) + ( 1 − a ) (5) \mathbf R(x) ={\mathbf L(x) − 1 + a\over (1 −{1\over a}+ \max_c {\mathbf L^c(x)\over a}+ \epsilon) }+(1 − a)\tag 5 R(x)=(1−a1+maxcaLc(x)+ϵ)L(x)−1+a+(1−a)(5)
we can see that when a = 1 a=1 a=1 the equation (5) and (2) reach the same result.
Exploit Structure of illumination to REFINE
min T ∥ T ^ − T ∥ F 2 + α ∥ W ⊙ ∇ T ∥ 1 (6) \min_ \mathbf T\|\hat {\mathbf T} − \mathbf T\|^2_F+ α\|\mathbf W\odot \nabla \mathbf T\|_1 \tag 6 Tmin∥T^−T∥F2+α∥W⊙∇T∥1(6)
where
- W \mathbf W W is the weight matrix
- ∇ T \nabla \mathbf T ∇T is the first order derivative filter
- contains ∇ h T ∇_h\mathbf T ∇hT (horizontal) and ∇ v T ∇_v\mathbf T ∇vT (vertical)
the optimal problem ( 6 ) (6) (6) can be solved by the [[Basic Knowledge/Optimization/alternating multiplier descent method|alternating direction minimization technique]]
Sped-Up Solver to Problem
it is obvious that
lim ϵ → 0 + ∑ x ∑ d ∈ { h , v } W d ( x ) ( ∇ d T ( x ) ) 2 ∣ ∇ d T ( x ) ∣ + ϵ = ∥ W ◦ ∇ T ∥ 1 \lim_{\epsilon \rightarrow 0^+}\sum_x\sum_{d∈\{h,v\}} {\mathbf W_d(x)(∇_d\mathbf T(x))^2\over |∇_d\mathbf T(x)| +\epsilon} = \|\mathbf W ◦ ∇\mathbf T\|_1 ϵ→0+limx∑d∈{h,v}∑∣∇dT(x)∣+ϵWd(x)(∇dT(x))2=∥W◦∇T∥1
so we can use ∑ x ∑ d ∈ { h , v } W d ( x ) ( ∇ d T ( x ) ) 2 ∣ ∇ d T ^ ( x ) ∣ + ϵ \sum_x\sum_{d∈\{h,v\}}{\mathbf W_d(x)(∇_d\mathbf T(x))^2\over |∇_d\mathbf {\hat T}(x)| +\epsilon} ∑x∑d∈{h,v}∣∇dT^(x)∣+ϵWd(x)(∇dT(x))2 to approximate ∥ W ◦ ∇ T ∥ 1 \|\mathbf W ◦ ∇\mathbf T\|_1 ∥W◦∇T∥1
the problem ( 6 ) (6) (6) can be written as
min T ∥ T ^ − T ∥ F 2 + α ∑ x ∑ d ∈ { h , v } W d ( x ) ( ∇ d T ( x ) ) 2 ∣ ∇ d T ^ ( x ) ∣ + ϵ (7) \min_ \mathbf T\|\hat {\mathbf T} − \mathbf T\|^2_F+ α \sum_x\sum_{d∈\{h,v\}}{\mathbf W_d(x)(∇_d\mathbf T(x))^2\over |∇_d\mathbf {\hat T}(x)| +\epsilon} \tag 7 Tmin∥T^−T∥F2+αx∑d∈{h,v}∑∣∇dT^(x)∣+ϵWd(x)(∇dT(x))2(7)
Thus, the problem can be directly obtained by solving the following
( I + α ∑ d ∈ { h , v } D d T Diag ( w ~ d ) D d ) t = t ^ (\mathbf I +\alpha \sum_{d∈\{h,v\}}\mathbf D^T_d \text {Diag}(\tilde{\mathbf w}_d)\mathbf D_d)\mathbf t = \hat {\mathbf t} (I+αd∈{h,v}∑DdTDiag(w~d)Dd)t=t^
where
- w ~ d \tilde {\mathbf w}_d w~d is the vectorized W d ( x ) ∣ ∇ d T ^ ( x ) ∣ + ϵ {\mathbf W_d(x)\over |∇_d\mathbf {\hat T}(x)| +\epsilon} ∣∇dT^(x)∣+ϵWd(x)
- D d T \mathbf D^T_d DdT is the Toeplitz matrices from the discrete gradient operators with forward difference in direction d d d
By doing this, the complexity reaches O ( N ) \mathcal O(N) O(N)
Possible Weighting Strategies
simply set to 1
W d ( x ) ← 1 , d ∈ { h , v } \mathbf W_d(x) ← \mathbf 1,d\in\{h,v\} Wd(x)←1,d∈{h,v}
set the entire coefficient to 1
W d ( x ) ← 1 ∣ ∇ d T ^ ( x ) ∣ + ϵ , d ∈ { h , v } (8) \mathbf W_d(x) ← {1\over |∇_d\mathbf {\hat T}(x)| +\epsilon },d\in\{h,v\}\tag 8 Wd(x)←∣∇dT^(x)∣+ϵ1,d∈{h,v}(8)
set by Relative Total Variation
W d ( x ) ← ∑ y ∈ Ω ( x ) G σ ( x , y ) ∣ ∑ y ∈ Ω ( x ) G σ ( x , y ) ∇ d T ^ ( y ) ∣ + ϵ , d ∈ { h , v } \mathbf W_d(x) ← \sum_{y∈\Omega(x)}{G_σ(x, y)\over |\sum_{y∈\Omega(x)}G_σ(x, y)∇_d\hat {\mathbf T}(y)| + \epsilon} ,d\in\{h,v\} Wd(x)←y∈Ω(x)∑∣∑y∈Ω(x)Gσ(x,y)∇dT^(y)∣+ϵGσ(x,y),d∈{h,v}
where
- G σ ( x , y ) ∝ exp ( − dist ( x , y ) 2 σ 2 ) G_σ(x, y)\propto \exp\left(-{\text{dist}(x,y)\over2\sigma^2}\right) Gσ(x,y)∝exp(−2σ2dist(x,y))
- dist ( x , y ) \text{dist}(x,y) dist(x,y) is to measure the spatial Euclidean distance between x x x and y y y
it is noticeable that when σ → 0 + \sigma\rightarrow 0^+ σ→0+, W \mathbf W W is as same as expression ( 8 ) (8) (8)
Other refine operations
Gamma correction
- set γ = 0.8 \gamma = 0.8 γ=0.8
- the following pics (a)-© show that γ = 0.5 , γ = 0.8 \gamma=0.5, \gamma=0.8 γ=0.5,γ=0.8 and γ = 1 \gamma=1 γ=1 respectively.
Denoising
- using BM3D to denoise
- execute BM3D on R \mathbf R R’s Y channel by converting R \mathbf R R to YUV
- alleviate the unbalance of processing
- dark places are well-denoised
- brighter places are over-smoothed
R f ← R ⊙ T + R d ⊙ ( 1 − T ) \mathbf R_f← \mathbf R \odot \mathbf T + \mathbf R_d \odot (1 − \mathbf T) Rf←R⊙T+Rd⊙(1−T)
where
- R d \mathbf R_d Rd is the BM3D denoising result
- R f \mathbf R_f Rf is the final recomposin
g result
this expression means that compose the denoised low-light patches ( 1 − T 1 − \mathbf T 1−T) and the un-denoised high-light patches ( T \mathbf T T)
the following pic (e) is the composed result of (b) and (d)
Highlights
- Sped-Up Solver to Problem reduce the complexity to O ( N ) \mathcal O(N) O(N)
- Possible Weighting Strategies set as Relative Total Variation, make W \mathbf W W more adaptive
- Denoising the image by composing denoised and un-denoised image