剪枝系列4：A Bayesian Optimization Framework for Neural Network Compression

这篇论文把贝叶斯优化用到了模型压缩中，提出了一个优化框架，具体压缩的方法可以灵活使用。超参数$\theta$表示最终的网络有多小，比如，当使用剪枝方法时它是一个threshold，当使用SVD是它是一个rank。

本文考虑的是如何选择压缩超参数$\theta$。根据BO方法，我们需要确定objective function和acquisition function。

Objective function

object function 需要考虑的是1.the quality of the compressed network。用$\mathcal{Q}({\tilde{f}}_{\theta })$ 代表模型的表现，用$\mathcal{L}({\tilde{f}}_{t}heta,f^{\ast })$代表保真度。2.the size of the obtained network。用$R({\tilde{f}}_{\theta },f^{\ast })$表示压缩比例。那么优化问题可以表示为：
$$\begin{gathered} argma{x}_{\theta }(\underset{J_Q(\theta )}{\gamma \mathcal{Q}({\tilde{f}}_{\theta })+R({\tilde{f}}_{\theta },f^{\ast }){)}^{-1}} \\ orargmi{n}_{\theta }(\underset{J_L(\theta )}{\kappa \mathcal{L}({\tilde{f}}_{\theta },f^{\ast })+R({\tilde{f}}_{\theta },f^{\ast }))} \end{gathered}$$
这里我们考虑用知识蒸馏的目标函数：
$$\mathcal{L}({\tilde{f}}_{\theta },f^{\ast }):={\mathbb{E}}_{x\sim P}(||{\tilde{f}}_{\theta }(x)-f^{\ast }(x)|{|}_2^2)=||f^{\ast }-{\tilde{f}}_{\theta }|{|}_2^2$$

acquisition function

Experiments

Comparison of different model selection methods on Resnet18

Knowledge distillation as a proxy for risk

一个自然的疑问是用知识蒸馏的目标函数（L2）是否在网络压缩中好用。实验结果表明用function norm 和用top-1 error rate 有相当的表现。

Compression of VGG-16

In this section, we demonstrate that our method finds compression parameters that compare favorably to state-ofthe-art compression results reported on VGG-16 [10]. We first apply our method to compress convolutional layers of VGG-16 using tensor decomposition, which has 13 parameters. After that, we fine-tune the compressed model for 5 epochs, using Stochastic Gradient Descent (SGD) with
momentum 0.9 and learning rate 1e-4, decreased by a factor of 10 every epoch. Second, we apply another pass of our algorithm to compress the fully-connected layers of the fine-tuned model using SVD, which has 3 parameters. A single optimization takes approximately 10 minutes. Again, after the compression, we fine-tune the compressed model…

Conclusion

In this work, we have developed a principled, fast, and flexible framework for optimizing neural network compression parameters…