gated convolution
1 Local response normalization（局部响应归一化）
- 局部响应归一化，由于做了平滑处理，可以增加泛化能力，公式如下
b x , y i = a x , y i / ( b i a s + α ∑ j = max ( 0 , i − n / 2 ) min ( N − 1 , i + n / 2 ) ( a x , y i ) 2 ) b_{x,y}^i = a_{x,y}^i/\left( {bias + \alpha \sum\limits_{j = \max (0,i - n/2)}^{\min (N - 1,i + n/2)} {{{(a_{x,y}^i)}^2}} } \right) bx,yi=ax,yi/⎝⎛bias+αj=max(0,i−n/2)∑min(N−1,i+n/2)(ax,yi)2⎠⎞
其中i代表在第几个feature map，(x,y)代表所在feature map位置 - tensorflow有内置函数tf.nn.lrn，参数输入与解释如下
"""Local Response Normalization.
The 4-D `input` tensor is treated as a 3-D array of 1-D vectors (along the last
dimension), and each vector is normalized independently. Within a given vector,
each component is divided by the weighted, squared sum of inputs within
`depth_radius`. In detail,
sqr_sum[a, b, c, d] =
sum(input[a, b, c, d - depth_radius : d + depth_radius + 1] ** 2)
output = input / (bias + alpha * sqr_sum) ** beta
a is batch size. d is channel.
2 gated convolution
公式如下
y ( X ) = ( X ∗ W + b ) ⊗ σ ( X ∗ V + c ) y({\bf{X}}) = ({\bf{X*W + b}}) \otimes \sigma ({\bf{X*V + c}}) y(X)=(X∗W+b)⊗σ(X∗V+c)
其中 W，V为两个不同的卷积核
tensorflow实现方法如下
def gate_conv(x_in, cnum, ksize, stride=1, rate=1, name='conv',
padding='SAME', activation='leaky_relu', use_lrn=True, training=True):
assert padding in ['SYMMETRIC', 'SAME', 'REFELECT']
if padding == 'SYMMETRIC' or padding == 'REFELECT':
p = int(rate * (ksize - 1) / 2)
x = tf.pad(x_in, [[0, 0], [p, p], [p, p], [0, 0]], mode=padding)
padding = 'VALID'
x = tf.layers.conv2d(
x_in, cnum, ksize, stride, dilation_rate=rate,
activation=None, padding=padding, name=name)
if use_lrn:
x = tf.nn.lrn(x, bias=0.00005)
if activation == 'leaky_relu':
x = tf.nn.leaky_relu(x)
g = tf.layers.conv2d(
x_in, cnum, ksize, stride, dilation_rate=rate,
activation=tf.nn.sigmoid, padding=padding, name=name + '_g')
x = tf.multiply(x, g)
return x, g