Paper Reading: Variational Convolutional Neural Network Pruning

A pruning method that based on Batch Normalization layer.
Variational Convolutional Neural Network Pruning

Brief introduction

To prune a filter in a convolutional layer, we should have a criterion that measures how this filter performs. In this paper, the author propose a new criterion called channel saliency which comes from batch normalization layer.

Implementation

What is channel saliency

Below is the implementation of batch normalization layer
$$BN(x)=\frac{x-{\mu }_B}{\sqrt{{\sigma }_B^2+ϵ}}$$
where ${\mu }_B$ and ${\sigma }_B^2$ is mean and variance of batch data, and $ϵ$ is a small constant(1e-5 or similar) that prevent zero occurs on denominator.

Then $x_{out}=\gamma BN(x)+\beta$ can be rewritten as
$$x_{out}=\gamma BN(x)+\tilde{\beta }$$
where $\tilde{\beta }=\gamma \beta$. And here we got
$$x_{out}=\gamma BN(x)+\tilde{\beta }=\gamma [BN(x)+\beta ]$$
The paramrter $\gamma$ here can be viewed as a criterion that measures importance of related filter directily. So we givethe parameter $\gamma$ a name called channel saliency.
Below, I just use $\gamma$ to represent channel saliency.

How to make good use of channel saliency parameter

It’s explict that we should just prune filters that have a small $\gamma$ in the following BN layer. The problem is that $\gamma$ changes drastically after several iterations in training stage, so the author analyzes the distribution of $\gamma$. When the distribution is around zero, then we can prune safely.

Meansuring distribution of channel saliency parameter

In an easy-understanding way

1. Using a new distribution $q_{\varphi }(\gamma )$ to estimate the distribution of $\gamma$ :$p(\gamma )$.
2. Create a new loss function:$$\mathcal{L}={\mathcal{L}}_d-KL(q_{\varphi }(\gamma )||p(\gamma ))$$where ${\mathcal{L}}_d$ is the original loss function, and KL is KL-divergenc in short.
3. KL-divergnce added to loss function can reduce the distance between $q_{\varphi }(\gamma )$ and $p(\gamma )$.
4. $p(\gamma )$ can be read from caculating result of a mini-batch, then we got $$p({\gamma }_i)\sim \mathcal{N}({\mu }_i,{\sigma }_i)$$where ${\mu }_i$ and ${\sigma }_i$ are what I need.
5. $q_{\varphi }(\gamma )$ is gnerated by us and $$q_{\varphi }({\gamma }_i)\sim \mathcal{N}({\mu }_i^{\ast },{\sigma }_i^{\ast })$$where ${\mu }_i^{\ast }$ and ${\sigma }_i^{\ast }$ are what we expect the network to learn.
   The author fix mean to zero and list out: $$q_{\varphi }({\gamma }_i)\sim \mathcal{N}(0,{\sigma }_i^{\ast })$$I do not think it right.
6. Then KL-divergence can be simplified as $$KL(q_{\varphi }(\gamma )||p(\gamma ))=\sum KL(q_{\varphi }({\gamma }_i)||p({\gamma }_i))=\sum [\frac{{\sigma }_i^{\ast }}{{\sigma }_i}+\frac{{\sigma }_i^{\ast }+({\mu }_i-{\mu }_i^{\ast }{)}^2}{2({\sigma }_i^{\ast }{)}^2}-\frac{1}{2}]$$
7. Finally, just backprograte loss function $\mathcal{L}$ and we can get estimated distribution of $\gamma$: ${\mu }_i^{\ast }$ and ${\sigma }_i^{\ast }$. When $({\mu }_i^{\ast },{\sigma }_i^{\ast })$ smaller then a threshold $({\mu }_T,{\sigma }_T)$, we just prune this filter.

In a detailed way

Introduction to variational inference

The general idea of VI(variational inference) is:

1. Define a flexible famliy of distributions over the hidden variables indexed by free parameters.
2. Find the setting of parameters that make the distribution family closest to the desired posterior distribution.
3. Thus the problem of finding distribution becomes a problem of optimization.

Let’s start from Bayes Rule
$$P(A_i|B)=\frac{P(B|A_i)P(A_i)}{{\sum }_{j}P(B|A_j)P(A_j)}$$
For a general problem, we want to find out the posterior distribution $p(z|x)$, in fact
$$p(z|x)=\frac{p(z)p(x|z)}{p(x)}$$
where $p(z)$ is the prior distribution and $p(x|z)$ is the likehood function. The problem is, however, $p(x)=\int p(z)p(x|z)$ is usually not computational-tractable. So VI method use another variational distribution $q_{\theta }(z)$ that depends on parameter $\theta$ to appoximate $p(z|x)$. Thus we apply the following function to optimize:
$$\underset{\theta }{\min }KL(q_{\theta }(z)||p(z,x))\text{\hspace{1em}(2.3.1)}$$
where $KL$ is Kullback–Leibler divergence(KL divergence).
The function above is hard to optimize because we don’t know $p(z,x)$. Then some transfer this function to:
$$\underset{\theta }{\max }ELBO(\theta ,x)={\mathbb{E}}_{q(z,\theta )}[\log \frac{p(x,z)}{q_{\theta }(z)}]\text{\hspace{1em}(2.3.2)}$$
ELBO here is Evidence Lower Bound in short. In many cases, $p(x,z)$ is our model in machine learning.
There exists a lot of methods that can optimize function $2.3.2$, including mean-field assumption, stochastic variational inference (SVI), black box variational inference (BBVI) , reparameterization tricks etc.

Apply variational inference to estimating channel saliency

In this paper, the concrete situation is
$$p(\gamma |\mathcal{D})=\frac{p(\gamma )p(\mathcal{D}|\gamma )}{p(\gamma )}$$
where $\mathcal{D}$ dataset. However, $p(\mathcal{D})=\int p(\gamma )p(\mathcal{D}|\gamma ))=\int p(\mathcal{D},\gamma )d\gamma$ is computational-intractable.
Contents for detailed is too borining and I don’t think it necessary to analyze. The author describe those manture technologies in detail, which is unnecessary. Just refer to easy-understanding way will be enough.

Experiment result

There exists results on CIFAR10, CIFAR100 and ImageNet2012 dataset, but I just list results on CIFAR10 and ImageNet2012 here.

CIFAR10 dataset

Result on VGG16 does not show much improvement. Result on ResNet differs quite a lot from my data in FLOPs. FLOPs shown here for original model is much smaller than my calculation, which makes it inconvincing in my eyes.

ImageNet2012 dataset

Experiments on ImageNet2012 does not achieve state-of-the-art result.

My conclusion

This method use bach normalization layer to determine which layer to prune. Experiments shown here can not achieve statre-of-the-art result, but the idea is worth rethinking.

Reference

[1] 变分推断（Variational Inference）最新进展简述
[2] 变分推断学习笔记
[3] ELBO 与 KL散度
[4] Stochastic Variational Inference
[5] 请解释下variational inference
[6] 变分推断中的ELBO(证据下界)