网络加速和压缩技术论文整理

从加速和压缩本身来说,两者不是同一件事,但通常情况下我们往往会同时做加速和压缩,两者都会给网络的计算带来收益,所以我们习惯将它们放在一起来讲。

低秩近似(low-rank Approximation),网络剪枝(network pruning),网络量化(network quantization),知识蒸馏(knowledge distillation)和紧凑网络设计(compact Network design)

低秩近似(low-rank Approximation)

网络剪枝(network pruning)

网络量化(network quantization)

紧凑网络设计(compact Network design)

知识蒸馏(knowledge distillation)

  • 2011-JMLR-Learning with Structured Sparsity
  • 2013-NIPS-Predicting Parameters in Deep Learning
  • 2014-BMVC-Speeding up convolutional neural networks with low rank expansions
  • 2014-NIPS-Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
  • 2014-NIPS-Do deep neural nets really need to be deep
  • 2015-ICML-Compressing neural networks with the hashing trick
  • 2015-INTERSPEECH-A Diversity-Penalizing Ensemble Training Method for Deep Learning
  • 2015-BMVC-Data-free parameter pruning for deep neural networks
  • 2015-NIPS-Learning both Weights and Connections for Efficient Neural Network
  • 2015_NIPSw-Distilling Intractable Generative Models
  • 2015-CVPR-Learning to generate chairs with convolutional neural networks
  • 2015-CVPR-Understanding deep image representations by inverting them [2016 IJCV version: Visualizing deep convolutional neural networks using natural pre-images]
  • 2015-CVPR-Efficient and Accurate Approximations of Nonlinear Convolutional Networks [2016 TPAMI version: Accelerating Very Deep Convolutional Networks for Classification and Detection]
  • 2016-ICLRb-Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
  • 2016-ICLR-All you need is a good init [Code]
  • 2016-ICLR-Convolutional neural networks with low-rank regularization
  • 2016-ICLR-Diversity networks
  • 2016-EMNLP-Sequence-Level Knowledge Distillation
  • 2016-CVPR-Inverting Visual Representations with Convolutional Networks
  • 2016-NIPS-Learning Structured Sparsity in Deep Neural Networks
  • 2016-NIPS-Dynamic Network Surgery for Efficient DNNs
  • 2016.10-Deep model compression: Distilling knowledge from noisy teachers
  • 2017-ICLR-Pruning Convolutional Neural Networks for Resource Efficient Inference
  • 2017-ICLR-Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
  • 2017-ICLR-Do Deep Convolutional Nets Really Need to be Deep and Convolutional?
  • 2017-ICML-Variational dropout sparsifies deep neural networks
  • 2017-CVPR-Learning deep CNN denoiser prior for image restoration
  • 2017-CVPR-Deep roots: Improving cnn efficiency with hierarchical filter groups
  • 2017-CVPR-All You Need is Beyond a Good Init: Exploring Better Solution for Training Extremely Deep Convolutional Neural Networks with Orthonormality and Modulation
  • 2017-CVPR-ResNeXt-Aggregated Residual Transformations for Deep Neural Networks
  • 2017-CVPR-Xception: Deep learning with depthwise separable convolutions
  • 2017-ICCV-Channel pruning for accelerating very deep neural networks [Code]
  • 2017-ICCV-Learning efficient convolutional networks through network slimming [Code]
  • 2017-ICCV-ThiNet: A filter level pruning method for deep neural network compression [Project]
  • 2017-ICCV-Interleaved group convolutions
  • 2017-NIPS-Net-trim: Convex pruning of deep neural networks with performance guarantee
  • 2017-NIPS-Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon
  • 2017-NNs-Nonredundant sparse feature extraction using autoencoders with receptive fields clustering
  • 2017.02-The Power of Sparsity in Convolutional Neural Networks
  • 2018-AAAI-Auto-balanced Filter Pruning for Efficient Convolutional Neural Networks
  • 2018-AAAI-Deep Neural Network Compression with Single and Multiple Level Quantization
  • 2018-ICML-On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization
  • 2018-ICMLw-Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures
  • 2018-ICLRo-Training and Inference with Integers in Deep Neural Networks
  • 2018-ICLR-Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers
  • 2018-ICLR-N2N learning: Network to Network Compression via Policy Gradient Reinforcement Learning
  • 2018-ICLR-Model compression via distillation and quantization
  • 2018-ICLR-Towards Image Understanding from Deep Compression Without Decoding
  • 2018-ICLR-Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
  • 2018-ICLR-Mixed Precision Training of Convolutional Neural Networks using Integer Operations
  • 2018-ICLR-Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy
  • 2018-ICLR-Loss-aware Weight Quantization of Deep Networks
  • 2018-ICLR-Alternating Multi-bit Quantization for Recurrent Neural Networks
  • 2018-ICLR-Adaptive Quantization of Neural Networks
  • 2018-ICLR-Variational Network Quantization
  • 2018-ICLR-Learning Sparse Neural Networks through L0 Regularization
  • 2018-ICLRw-To Prune, or Not to Prune: Exploring the Efficacy of Pruning for Model Compression (Similar topic: 2018-NIPSw-nip in the bud, 2018-NIPSw-rethink)
  • 2018-ICLRw-Systematic Weight Pruning of DNNs using Alternating Direction Method of Multipliers
  • 2018-ICLRw-Weightless: Lossy weight encoding for deep neural network compression
  • 2018-ICLRw-Variance-based Gradient Compression for Efficient Distributed Deep Learning
  • 2018-ICLRw-Stacked Filters Stationary Flow For Hardware-Oriented Acceleration Of Deep Convolutional Neural Networks
  • 2018-ICLRw-Training Shallow and Thin Networks for Acceleration via Knowledge Distillation with Conditional Adversarial Networks
  • 2018-ICLRw-Accelerating Neural Architecture Search using Performance Prediction
  • 2018-ICLRw-Nonlinear Acceleration of CNNs
  • 2018-CVPR-Context-Aware Deep Feature Compression for High-Speed Visual Tracking
  • 2018-CVPR-NISP: Pruning Networks using Neuron Importance Score Propagation
  • 2018-CVPR-“Learning-Compression” Algorithms for Neural Net Pruning
  • 2018-CVPR-Deep Image Prior [Code]
  • 2018-CVPR-Condensenet: An efficient densenet using learned group convolutions
  • 2018-CVPR-Shift: A zero flop, zero parameter alternative to spatial convolutions
  • 2018-CVPR-Interleaved structured sparse convolutional neural networks
  • 2018-IJCAI-Efficient DNN Neuron Pruning by Minimizing Layer-wise Nonlinear Reconstruction Error
  • 2018-IJCAI-Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks
  • 2018-IJCAI-Where to Prune: Using LSTM to Guide End-to-end Pruning
  • 2018-IJCAI-Accelerating Convolutional Networks via Global & Dynamic Filter Pruning
  • 2018-IJCAI-Optimization based Layer-wise Magnitude-based Pruning for DNN Compression
  • 2018-IJCAI-Progressive Blockwise Knowledge Distillation for Neural Network Acceleration
  • 2018-IJCAI-Complementary Binary Quantization for Joint Multiple Indexing
  • 2018-ECCV-A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers
  • 2018-ECCV-Coreset-Based Neural Network Compression
  • 2018-ECCV-Data-Driven Sparse Structure Selection for Deep Neural Networks [Code]
  • 2018-BMVCo-Structured Probabilistic Pruning for Convolutional Neural Network Acceleration
  • 2018-BMVC-Efficient Progressive Neural Architecture Search
  • 2018-BMVC-Igcv3: Interleaved lowrank group convolutions for efficient deep neural networks
  • 2018-NIPS-Discrimination-aware Channel Pruning for Deep Neural Networks
  • 2018-NIPS-Frequency-Domain Dynamic Pruning for Convolutional Neural Networks
  • 2018-NIPS-ChannelNets: Compact and Efficient Convolutional Neural Networks via Channel-Wise Convolutions
  • 2018-NIPS-DropBlock: A regularization method for convolutional networks
  • 2018-NIPS-Constructing fast network through deconstruction of convolution
  • 2018-NIPS-Learning Versatile Filters for Efficient Convolutional Neural Networks [Code]
  • 2018-NIPSw-Pruning neural networks: is it time to nip it in the bud?
  • 2018-NIPSwb-Rethinking the Value of Network Pruning [2019 ICLR version]
  • 2018-NIPSw-Structured Pruning for Efficient ConvNets via Incremental Regularization
  • 2018.05-Compression of Deep Convolutional Neural Networks under Joint Sparsity Constraints
  • 2018.05-AutoPruner: An End-to-End Trainable Filter Pruning Method for Efficient Deep Model Inference
  • 2018.11-Second-order Optimization Method for Large Mini-batch: Training ResNet-50 on ImageNet in 35 Epochs
  • 2018.11-Rethinking ImageNet Pre-training (Kaiming He)
  • 2019-ICLRo-The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
  • 2019-AAAIo-A layer decomposition-recomposition framework for neuron pruning towards accurate lightweight networks
  • 2019-CVPR-All You Need is a Few Shifts: Designing Efficient Convolutional Neural Networks for Image Classification
  • 2019-CVPR-HetConv Heterogeneous Kernel-Based Convolutions for Deep CNNs
  • 2019-CVPR-Fully Learnable Group Convolution for Acceleration of Deep Neural Networks
  • 2019-CVPR-Towards Optimal Structured CNN Pruning via Generative Adversarial Learning
  • 2019-CVPR-Centripetal SGD for Pruning Very Deep Convolutional Networks with Complicated Structure
  • 2019-BigComp-Towards Robust Compressed Convolutional Neural Networks
  • 2019-PR-Filter-in-Filter: Improve CNNs in a Low-cost Way by Sharing Parameters among the Sub-filters of a Filter
  • 2019-PRL-BDNN: Binary Convolution Neural Networks for Fast Object Detection
  • 2019.03-MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning (Face++)
  • 2019.04-Data-Free Learning of Student Networks (Huawei)
  • 2019.04-Resource Efficient 3D Convolutional Neural Networks
  • 2019.04-Meta Filter Pruning to Accelerate Deep Convolutional Neural Networks
  • 2019.04-Knowledge Squeezed Adversarial Network Compression- 2011-JMLR-Learning with Structured Sparsity
  • 2013-NIPS-Predicting Parameters in Deep Learning
  • 2014-BMVC-Speeding up convolutional neural networks with low rank expansions
  • 2014-NIPS-Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
  • 2014-NIPS-Do deep neural nets really need to be deep
  • 2015-ICML-Compressing neural networks with the hashing trick
  • 2015-INTERSPEECH-A Diversity-Penalizing Ensemble Training Method for Deep Learning
  • 2015-BMVC-Data-free parameter pruning for deep neural networks
  • 2015-NIPS-Learning both Weights and Connections for Efficient Neural Network
  • 2015_NIPSw-Distilling Intractable Generative Models
  • 2015-CVPR-Learning to generate chairs with convolutional neural networks
  • 2015-CVPR-Understanding deep image representations by inverting them [2016 IJCV version: Visualizing deep convolutional neural networks using natural pre-images]
  • 2015-CVPR-Efficient and Accurate Approximations of Nonlinear Convolutional Networks [2016 TPAMI version: Accelerating Very Deep Convolutional Networks for Classification and Detection]
  • 2016-ICLRb-Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
  • 2016-ICLR-All you need is a good init [Code]
  • 2016-ICLR-Convolutional neural networks with low-rank regularization
  • 2016-ICLR-Diversity networks
  • 2016-EMNLP-Sequence-Level Knowledge Distillation
  • 2016-CVPR-Inverting Visual Representations with Convolutional Networks
  • 2016-NIPS-Learning Structured Sparsity in Deep Neural Networks
  • 2016-NIPS-Dynamic Network Surgery for Efficient DNNs
  • 2016.10-Deep model compression: Distilling knowledge from noisy teachers
  • 2017-ICLR-Pruning Convolutional Neural Networks for Resource Efficient Inference
  • 2017-ICLR-Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
  • 2017-ICLR-Do Deep Convolutional Nets Really Need to be Deep and Convolutional?
  • 2017-ICML-Variational dropout sparsifies deep neural networks
  • 2017-CVPR-Learning deep CNN denoiser prior for image restoration
  • 2017-CVPR-Deep roots: Improving cnn efficiency with hierarchical filter groups
  • 2017-CVPR-All You Need is Beyond a Good Init: Exploring Better Solution for Training Extremely Deep Convolutional Neural Networks with Orthonormality and Modulation
  • 2017-CVPR-ResNeXt-Aggregated Residual Transformations for Deep Neural Networks
  • 2017-CVPR-Xception: Deep learning with depthwise separable convolutions
  • 2017-ICCV-Channel pruning for accelerating very deep neural networks [Code]
  • 2017-ICCV-Learning efficient convolutional networks through network slimming [Code]
  • 2017-ICCV-ThiNet: A filter level pruning method for deep neural network compression [Project]
  • 2017-ICCV-Interleaved group convolutions
  • 2017-NIPS-Net-trim: Convex pruning of deep neural networks with performance guarantee
  • 2017-NIPS-Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon
  • 2017-NNs-Nonredundant sparse feature extraction using autoencoders with receptive fields clustering
  • 2017.02-The Power of Sparsity in Convolutional Neural Networks
  • 2018-AAAI-Auto-balanced Filter Pruning for Efficient Convolutional Neural Networks
  • 2018-AAAI-Deep Neural Network Compression with Single and Multiple Level Quantization
  • 2018-ICML-On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization
  • 2018-ICMLw-Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures
  • 2018-ICLRo-Training and Inference with Integers in Deep Neural Networks
  • 2018-ICLR-Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers
  • 2018-ICLR-N2N learning: Network to Network Compression via Policy Gradient Reinforcement Learning
  • 2018-ICLR-Model compression via distillation and quantization
  • 2018-ICLR-Towards Image Understanding from Deep Compression Without Decoding
  • 2018-ICLR-Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
  • 2018-ICLR-Mixed Precision Training of Convolutional Neural Networks using Integer Operations
  • 2018-ICLR-Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy
  • 2018-ICLR-Loss-aware Weight Quantization of Deep Networks
  • 2018-ICLR-Alternating Multi-bit Quantization for Recurrent Neural Networks
  • 2018-ICLR-Adaptive Quantization of Neural Networks
  • 2018-ICLR-Variational Network Quantization
  • 2018-ICLR-Learning Sparse Neural Networks through L0 Regularization
  • 2018-ICLRw-To Prune, or Not to Prune: Exploring the Efficacy of Pruning for Model Compression (Similar topic: 2018-NIPSw-nip in the bud, 2018-NIPSw-rethink)
  • 2018-ICLRw-Systematic Weight Pruning of DNNs using Alternating Direction Method of Multipliers
  • 2018-ICLRw-Weightless: Lossy weight encoding for deep neural network compression
  • 2018-ICLRw-Variance-based Gradient Compression for Efficient Distributed Deep Learning
  • 2018-ICLRw-Stacked Filters Stationary Flow For Hardware-Oriented Acceleration Of Deep Convolutional Neural Networks
  • 2018-ICLRw-Training Shallow and Thin Networks for Acceleration via Knowledge Distillation with Conditional Adversarial Networks
  • 2018-ICLRw-Accelerating Neural Architecture Search using Performance Prediction
  • 2018-ICLRw-Nonlinear Acceleration of CNNs
  • 2018-CVPR-Context-Aware Deep Feature Compression for High-Speed Visual Tracking
  • 2018-CVPR-NISP: Pruning Networks using Neuron Importance Score Propagation
  • 2018-CVPR-“Learning-Compression” Algorithms for Neural Net Pruning
  • 2018-CVPR-Deep Image Prior [Code]
  • 2018-CVPR-Condensenet: An efficient densenet using learned group convolutions
  • 2018-CVPR-Shift: A zero flop, zero parameter alternative to spatial convolutions
  • 2018-CVPR-Interleaved structured sparse convolutional neural networks
  • 2018-IJCAI-Efficient DNN Neuron Pruning by Minimizing Layer-wise Nonlinear Reconstruction Error
  • 2018-IJCAI-Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks
  • 2018-IJCAI-Where to Prune: Using LSTM to Guide End-to-end Pruning
  • 2018-IJCAI-Accelerating Convolutional Networks via Global & Dynamic Filter Pruning
  • 2018-IJCAI-Optimization based Layer-wise Magnitude-based Pruning for DNN Compression
  • 2018-IJCAI-Progressive Blockwise Knowledge Distillation for Neural Network Acceleration
  • 2018-IJCAI-Complementary Binary Quantization for Joint Multiple Indexing
  • 2018-ECCV-A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers
  • 2018-ECCV-Coreset-Based Neural Network Compression
  • 2018-ECCV-Data-Driven Sparse Structure Selection for Deep Neural Networks [Code]
  • 2018-BMVCo-Structured Probabilistic Pruning for Convolutional Neural Network Acceleration
  • 2018-BMVC-Efficient Progressive Neural Architecture Search
  • 2018-BMVC-Igcv3: Interleaved lowrank group convolutions for efficient deep neural networks
  • 2018-NIPS-Discrimination-aware Channel Pruning for Deep Neural Networks
  • 2018-NIPS-Frequency-Domain Dynamic Pruning for Convolutional Neural Networks
  • 2018-NIPS-ChannelNets: Compact and Efficient Convolutional Neural Networks via Channel-Wise Convolutions
  • 2018-NIPS-DropBlock: A regularization method for convolutional networks
  • 2018-NIPS-Constructing fast network through deconstruction of convolution
  • 2018-NIPS-Learning Versatile Filters for Efficient Convolutional Neural Networks [Code]
  • 2018-NIPSw-Pruning neural networks: is it time to nip it in the bud?
  • 2018-NIPSwb-Rethinking the Value of Network Pruning [2019 ICLR version]
  • 2018-NIPSw-Structured Pruning for Efficient ConvNets via Incremental Regularization
  • 2018.05-Compression of Deep Convolutional Neural Networks under Joint Sparsity Constraints
  • 2018.05-AutoPruner: An End-to-End Trainable Filter Pruning Method for Efficient Deep Model Inference
  • 2018.11-Second-order Optimization Method for Large Mini-batch: Training ResNet-50 on ImageNet in 35 Epochs
  • 2018.11-Rethinking ImageNet Pre-training (Kaiming He)
  • 2019-ICLRo-The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
  • 2019-AAAIo-A layer decomposition-recomposition framework for neuron pruning towards accurate lightweight networks
  • 2019-CVPR-All You Need is a Few Shifts: Designing Efficient Convolutional Neural Networks for Image Classification
  • 2019-CVPR-HetConv Heterogeneous Kernel-Based Convolutions for Deep CNNs
  • 2019-CVPR-Fully Learnable Group Convolution for Acceleration of Deep Neural Networks
  • 2019-CVPR-Towards Optimal Structured CNN Pruning via Generative Adversarial Learning
  • 2019-CVPR-Centripetal SGD for Pruning Very Deep Convolutional Networks with Complicated Structure
  • 2019-BigComp-Towards Robust Compressed Convolutional Neural Networks
  • 2019-PR-Filter-in-Filter: Improve CNNs in a Low-cost Way by Sharing Parameters among the Sub-filters of a Filter
  • 2019-PRL-BDNN: Binary Convolution Neural Networks for Fast Object Detection
  • 2019.03-MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning (Face++)
  • 2019.04-Data-Free Learning of Student Networks (Huawei)
  • 2019.04-Resource Efficient 3D Convolutional Neural Networks
  • 2019.04-Meta Filter Pruning to Accelerate Deep Convolutional Neural Networks
  • 2019.04-Knowledge Squeezed Adversarial Network Compression

你可能感兴趣的:(Paper,Reading,Deep,Learning)