深度学习模型压缩分类及原理方法

深度学习模型压缩分类及原理方法

深度学习模型压缩分类

### 精细模型设计

### 模型裁剪及权重共享复用

### 低秩近似

### 知识蒸馏

### 核的稀疏化

### 模型量化（网络二值化）

剪枝

就是通过去除网络中冗余的channels,filters, neurons, or
layers以得到一个更轻量级的网络，同时不影响性能。

代表性的工作有：

奇异值分解SVD(NIPS 2014)：Exploiting linear structure within convolutional
networks for efficient evaluation

韩松(ICLR 2016)：Deep compression: Compressing deep neural networks with
pruning, trained quantization and huffman coding

(NIPS 2015)：Learning both weights and connections for efficient neural
network

频域压缩(NIPS 2016)：Packing convolutional neural networks in the frequency
domain

剪Filter Reconstruction Error(ICCV 2017)：Thinet: A filter level pruning method
for deep neural network compression

LASSO regression(ICCV 2017)：Channel pruning for accelerating very deep neural
networks

Discriminative channels(NIPS 2018)：Discrimination-aware channel pruning for
deep neural networks

剪枝(ICCV 2017)：Channel pruning for accelerating very deep neural networks

neuron level sparsity(ECCV 2017)：Less is more: Towards compact cnns

Structured Sparsity Learning(NIPS 2016)：Learning structured sparsity in deep
neural networks

蒸馏

蒸馏就是过模仿教师网络生成的软标签将知识从大的，预训练过的教师模型转移到轻量级的学生模型

代表性的工作有：

Hinton(NIPS 2015)：Distilling the knowledge in a neural network

中间层的特征作为提示(ICLR 2015)：Fitnets: Hints for thin deep nets

多个Teacher(SIGKDD 2017)：Learning from multiple teacher networks

两个特征得新知识再transfer(CVPR 2017)：A gift from knowledge distillation: Fast
optimization, network minimization and transfer learning

量化就是减少权重等的表示的位数，比如原来网络权值用32 bit存储，现在我只用8
bit来存储，以减少模型的Memory为原来的 [公式]
；更有甚者使用二值神经网络。代表性的工作有：

量化：

(ICML 2015)：Compressing neural networks with the hashing trick

(NIPS 2015)：Learning both weights and connections for efficient neural
network

(CVPR 2018)：Quantization and training of neural networks for efficient
integer-arithmetic-only inference

(CVPR 2016)：Quantized convolutional neural networks for mobile devices

(ICML 2018)：Deep k-means: Re-training and parameter sharing with harder
cluster assignments for compressing deep convolutions

(CVPR 2019)：Learning to quantize deep networks by optimizing quantization
intervals with task loss

(CVPR 2019)：HAQ: Hardware-Aware automated quantization with mixed precision.

二值神经网络：

Binarized weights(NIPS 2015)：BinaryConnect: Training deep neural networks with
binary weights during propagations

Binarized activations(NIPS 2016)：Binarized neural networks

XNOR(ECCV 2016)：Xnor-net: Imagenet classification using binary convolutional
neural networks

more weight and activation(NIPS 2017)：Towards accurate binary convolutional
neural network

(ECCV 2020)：Learning Architectures for Binary Networks

(ECCV 2020)：BATS: Binary ArchitecTure Search

低秩分解

低秩分解就是将原来大的权重矩阵分解成多个小的矩阵，而小矩阵的计算量都比原来大矩阵的计算量要小。代表性的工作有：

低秩分解(ICCV 2017)：On compressing deep models by low rank and sparse
decomposition

乐高网络(ICML 2019)：Legonet: Efficient convolutional neural networks with lego
filters

奇异值分解(NIPS 2014)：Exploiting Linear Structure Within Convolutional
Networks for Efficient Evaluation

轻量化模块设计

轻量化模块设计就是设计一些计算效率高，适合在端侧设备上部署的模块。代表性的工作有：

Bottleneck(ICLR 2017)：Squeezenet: Alexnet-level accuracy with 50x fewer
parameters and 0.5 mb model size

MobileNet(CVPR 2017)：Mobilenets: Efficient convolutional neural networks for
mobile vision applications

ShuffleNet(CVPR 2018)：Shufflenet: An extremely efficient convolutional neural
network for mobile devices

SE模块(CVPR 2018)：Squeeze-and-excitation networks

无参数的Shift操作(CVPR 2018)：Shift: A zero flop, zero parameter alternative to
spatial convolutions

Shift操作填坑(Arxiv)：Shift-based primitives for efficientconvolutional neural
networks

多用卷积核(NIPS 2018)：Learning versatile filters for efficient convolutional
neural networks

GhostNet(CVPR 2020)：GhostNet: More features from cheap operations

滤波器剪枝SOTA效果

基于滤波器骨架的逐条剪枝算法，刷新滤波器剪枝的SOTA效果；

https://github.com/fxmeng/Pruning-Filter-in-Filter

**6
加法网络就是：利用卷积所计算的互相关性其实就是一种“相似性的度量方法”，所以在神经网络中用加法代替乘法，在减少运算量的同时获得相同的性能。**代表性的工作有：

**(CVPR 20)：**AdderNet: Do We Really Need Multiplications in Deep Learning?
**(NIPS 20)：**Kernel Based Progressive Distillation for Adder Neural Networks
**(Arxiv)：**AdderSR: Towards Energy Efficient Image Super-Resolution

Slimming：

(ICCV 2017)：清华张长水，黄高团队：Learning Efficient Convolutional Networks
through Network Slimming

(ECCV 2020)：得克萨斯大学奥斯汀分校团队：GAN Slimming: All-in-One GAN
Compression by A Unified Optimization Framework

参考资料

https://github.com/topics/pytorch-deep-compression

https://www.cocopie.ai/

腾讯案例应用

https://cloud.tencent.com/developer/article/1593211

成熟应用

https://www.cocopie.ai/

https://www.volcengine.cn/