Image Inpainting for Irregular Holes Using Partial Convolutions理解与电镜图像应用

Image Inpainting for Irregular Holes Using Partial Convolutions

Partial Convolutional Layer

只对non-hole的部分做卷积运算

可以理解为对mask做卷积运算，卷积核的权值都为1，相当于对slide window对应的mask小区域求和，也就是公式中的sum(M)，如果结果大于0，该小区域卷积之后的结果为1，otherwise为0。这就做到了mask的更新只要网络深度足够大，mask最终都为1。

Network Architecture

## Loss Functions

These are the L 1 losses on the network output for the hole and the non-hole pixels respectively.

The perceptual loss computes the L 1 distances between both I out and I comp and the ground truth, but after projecting these images into higher level feature spaces using an ImageNet-pretrained VGG-16. Ψ p I ∗ is the activation map of the pth selected layer given original input I ∗ . We use layers pool1, pool2 and pool3 for our loss.

We further include the style-loss term, which is similar to the perceptual loss, but we first perform an autocorrelation (Gram matrix) on each feature map before applying the L 1 .

Our final loss term is the total variation (TV) loss L tv : which is the smoothing penalty on R, where R is the region of 1-pixel dilation of the hole region.

Experiments

Cremi数据集中存在几张带黑线的图像，这些黑线是由于切片时，不小心将部分细胞划破导致的，如果不将黑线去除，会影响到细胞分割的效果。

Trainset

Results

代码

https://github.com/MathiasGruber/PConv-Keras