Global Sensing and Measurements Reuse for Image Compressed Sensing

Global Sensing and Measurements Reuse for Image Compressed Sensing

Abstract

现有方法仅仅是从网络中的部分特征获得测量值，并且将其用于图像重建一次。

【Low-level feature： 通常是指图像中的一些小的细节信息，例如边缘（edge),角(corner),颜色(color),像素(pixeles), 梯度(gradients)等，这些信息可以通过滤波器、SIFT或HOG获取；
High level feature：是建立在low level feature之上的，可以用于图像中目标或物体形状的识别和检测，具有更丰富的语义信息。】

所以提出了MR-CCSNet(Measurements Reuse Convolutional Compressed Sensing Network)，其中GSM(Global Sensing Module)用于提取所有特征，MRB(Measurements Reuse Block)用于多次重建。

Introduction

GSM：

1. 用卷积层获得高维特征
2. 通过多个卷积层逐步压缩特征图
3. 收集网络中所有级别特征
4. 通过$1\times 1$卷积获得测量值

为了匹配维度，将池层添加到快捷连接中。

MRB：

1. 压缩相位重建(phased reconstructed)结果并获得多个特征图
2. 从测量中提取匹配信息
3. 在多尺度上融合它们

实验数据集：BSDS500[2]、Set5[4]、Set14[39]

评估指标：PSNR、SSIM

消融实验：GSM和MRB是有效的

贡献：

1. GSM可以实现有效的采样
2. MRB：充分利用测量
3. 构建了基于GSM和MRB的图像CS端到端MR-CSSNet，并测试了有效性

Related work

Traditional Compressed Sensing

即解决sparsity-regularized optimization problem
$$\underset{x}{\min }\frac{1}{2}\Vert \Phi x-y{\Vert }_2^2+\lambda \Vert \Psi x{\Vert }_1$$
$\Psi x$是$x$相对于域$\Psi$的变换系数，$\Psi x$的稀疏性由$1$范数表示

Deep Compressed Sensing

即用神经网络来求解逆问题，损失函数为
$$\underset{\theta }{\min }\frac{1}{2}\sum _{i=1}^k\Vert x_i-F(y_i,\theta ){\Vert }_2^2$$
$x_i$是原始图像，$y_i$是观测，$F$是神经网络，$\theta$是参数

[37]中是LAPRAN，来通过不同分辨率的多个阶段同时重建原始图像。

Methodology

采样率：6.25%

MR-CCSNet：【a sensing network GSM】+【an initial reconstruction network】+【a deep reconstruction network】

1. Obtain the measurements $y$ from the sensing network
2. The initial reconstruction network generates initial reconstructed image by a linear mapping
3. Refine the initial reconstructed image by a non-linear deep reconstruction network【initial reconstructed image is not enough】
4. 【stack multiple MRBs in the deep reconstruction network】

Sensing network $S(\cdot )$（GSM?）：

1. Use convolutional layers for the whole images.【直接对整个图像进行卷积，而不是将图像划分为非重叠块】
2. There is no bias and activation function.【为了满足线性特性】
3. $y=S(x)$，$x\in {\mathbb{R}}^{1\times H\times W}$，$y\in {\mathbb{R}}^{4\times \frac{H}{8}\times \frac{W}{8}}$

Initial reconstruction network $I(\cdot )$：the first time to utilize the measurements

1. Depth-wise convolution layer expands the measurements in channel dimension and the shape of channel becomes $64\times \frac{H}{8}\times \frac{W}{8}$.
2. Get a $1\times H\times W$ tensor by a pixel shuffle layer.

【pixel shuffle layer：一种对低分辨率特征图上采样的思路，假设打算将$H\times W\times C$的特征图在长和宽的维度上扩大$r$倍变成$rH\times rW\times C$，则通过深度为$r^{2}C$的卷积对$H\times W\times C$的特征图进行卷积得到$H\times W\times r^{2}C$的特征图，再通过“周期洗牌”的操作变成$rH\times rW\times C$】

【pixel shuffle layer就是输入$H\times W$的低分辨率的图像，输出$rH\times rW$的高分辨率的图像】

Deep reconstruction network $D(\cdot )$：the second time to utilize the measurements

1. Convert the initial reconstructed $I(y)$ image to a high dimensional feature by a convolutional layer.
2. Repeat MRBs.
3. Fuse the MRBs with matching features extracted from measurements $y$ multiple times on multi-scale.【用于将它们与多尺度上多次从测量$y$中提取的匹配特征融合】

Finally：

1. Use a convolutional layer to reconstruct the image from high dimensional features.
2. Add a shortcut connection to the deep reconstruction network.【shortcut connection是快捷连接，用于解决网络退化问题】

The final reconstructed image $\hat{x}$：
$$\hat{x}=D(I(y))+I(y)$$

GSM

【卷积神经网络以分层方式提取特征，则靠近输入的层学习低级特征，如线条和简单纹理。而深的层学习高级特征，如形状】

1. First stage：Use $3\times 3$ convolution layers to extract features.
2. Second stage：Collect all level features in the network.
3. Use a $1\times 1$ convolution layer to sample, rather than only from the low features (i.e. CSNet+) or high features (i.e. RK-CCSNet).

【To collect all level features for sampling, we use a shortcut connection to pass the features of different layers to the end, and the pooling layer is added for matching the dimensions.】

【采样率变化时，GSM不能很好的适配，故提出GSM+】

GSM+

Different from GSM：

1. Add a shortcut connection between two successive layers rather than add it from different layers to the end directly.

2. The building block of GSM+：
   $$y_{t+1}=Conv(y_t)+P(y_t)$$
   Conv and P denote convolution layer and meanpooling layer.

3. The sampling ratio is controlled by the number of building block and the blue block.

When the sampling ratio is $50\%$, there is only one building block in GSM+, so GSM+ degenerate into GSM.

MRB

Phased recontructed result $f_t\in {\mathbb{R}}^{C\times H\times W}$ and measurements $y\in {\mathbb{R}}^{C\times \frac{H}{4}\times \frac{W}{4}}$ are fed into MRB.

1. Use two convolutional layers, denoted as $Con{v}_1$ and $Con{v}_2$, to obtain a compacted feature map $f^{\downarrow }\in {\mathbb{R}}^{C\times \frac{H}{2}\times \frac{W}{2}}$ and $f^{\downarrow \downarrow }\in {\mathbb{R}}^{C\times \frac{H}{4}\times \frac{W}{4}}$.

2. $$\begin{gathered} f^{\downarrow }=Con{v}_1(f_t), \\ {f}^{\downarrow \downarrow }=Con{v}_2(f^{\downarrow }). \end{gathered}$$

3. Fig.5 extract matching information from measurements and obtain three feature maps $y_1\in {\mathbb{R}}^{C\times \frac{H}{4}\times \frac{W}{4}}$, $y_2\in {\mathbb{R}}^{C\times \frac{H}{2}\times \frac{W}{2}}$ and $y_3\in {\mathbb{R}}^{C\times H\times W}$ by Multi-Scale Reusing.

1. $F_1=Con{v}_3(f^{\downarrow \downarrow }\oplus y_1)$, in the third block. Then copy the $f^{\downarrow \downarrow }$ and fuse them with $F_1$.【怎么fuse的】
2. 接着pixel shuffle + Conv，即$f^{\uparrow }=Pixel(Con{v}_4(F_1\oplus f^{\downarrow \downarrow }))$，接着与$y_2$进行上述步骤。（$F_1\in {\mathbb{R}}^{C\times \frac{H}{4}\times \frac{W}{4}}$, $f^{\uparrow }\in {\mathbb{R}}^{C\times \frac{H}{2}\times \frac{W}{2}}$）后面依次下去，得到$f_{t+1}\in {\mathbb{R}}^{C\times H\times W}$

对于这块的channel融合不是很理解，如果有大佬明白的话，希望给我讲一下。

Loss function

For the initial reconstruction network, $l_{int}={\sum }_{k=1}^n\Vert I(S(y_k;\theta );{\varphi }_{int})-x_k{\Vert }_F^2$.

For the deep reconstruction network, $l_{deep}={\sum }_{k=1}^n\Vert D(I(S(y_k;\theta );{\varphi }_{int});{\varphi }_{deep})-x_k{\Vert }_F^2$

$\theta$, ${\varphi }_{int}$, ${\varphi }_{deep}$ denote the parameters of $S(\cdot )$, $I(\cdot )$ and $D(\cdot )$

The loss of MR-CCSNet is $l=l_{deep}+l_{int}$.

Experiments

Training datasets: 400 images from BSDS500[2]

Three standard benchmark datasets: Set5[4], Set14[39], BSDS100[2]

Conclusion

Future work

In the sensing network, pooling operation loses information about the low-level features.

Attention mechanism can effectively help us in extracting matching features from measurements.

In the real-world, because there are noise in the measurements, using them multiple times will introduce noise in the reconstruction process.
ching features from measurements.

In the real-world, because there are noise in the measurements, using them multiple times will introduce noise in the reconstruction process.