网络加速和压缩技术论文整理

从加速和压缩本身来说，两者不是同一件事，但通常情况下我们往往会同时做加速和压缩，两者都会给网络的计算带来收益，所以我们习惯将它们放在一起来讲。

低秩近似（low-rank Approximation），网络剪枝（network pruning），网络量化（network quantization），知识蒸馏（knowledge distillation）和紧凑网络设计（compact Network design）

低秩近似（low-rank Approximation）

网络剪枝（network pruning）

网络量化（network quantization）

• 2011-JMLR-​​Learning with Structured Sparsity​​
• 2013-NIPS-​​Predicting Parameters in Deep Learning​​
• 2014-BMVC-​​Speeding up convolutional neural networks with low rank expansions​​
• 2014-NIPS-​​Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation​​
• 2014-NIPS-​​Do deep neural nets really need to be deep​​
• 2015-ICML-​​Compressing neural networks with the hashing trick​​
• 2015-INTERSPEECH-​​A Diversity-Penalizing Ensemble Training Method for Deep Learning​​
• 2015-BMVC-​​Data-free parameter pruning for deep neural networks​​
• 2015-NIPS-​​Learning both Weights and Connections for Efficient Neural Network​​
• 2015_NIPSw-​​Distilling Intractable Generative Models​​
• 2015-CVPR-​​Learning to generate chairs with convolutional neural networks​​
• 2015-CVPR-​​Understanding deep image representations by inverting them​​​ [2016 IJCV version:​​Visualizing deep convolutional neural networks using natural pre-images​​]
• 2015-CVPR-​​Efficient and Accurate Approximations of Nonlinear Convolutional Networks​​​ [2016 TPAMI version:​​Accelerating Very Deep Convolutional Networks for Classification and Detection​​]
• 2016-ICLRb-​​Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding​​
• 2016-ICLR-​​All you need is a good init​​​ [​​Code​​]
• 2016-ICLR-​​Convolutional neural networks with low-rank regularization​​
• 2016-ICLR-​​Diversity networks​​
• 2016-EMNLP-​​Sequence-Level Knowledge Distillation​​
• 2016-CVPR-​​Inverting Visual Representations with Convolutional Networks​​
• 2016-NIPS-​​Learning Structured Sparsity in Deep Neural Networks​​
• 2016-NIPS-​​Dynamic Network Surgery for Efficient DNNs​​
• 2016.10-​​Deep model compression: Distilling knowledge from noisy teachers​​
• 2017-ICLR-​​Pruning Convolutional Neural Networks for Resource Efficient Inference​​
• 2017-ICLR-​​Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights​​
• 2017-ICLR-​​Do Deep Convolutional Nets Really Need to be Deep and Convolutional?​​
• 2017-ICML-​​Variational dropout sparsifies deep neural networks​​
• 2017-CVPR-​​Learning deep CNN denoiser prior for image restoration​​
• 2017-CVPR-​​Deep roots: Improving cnn efficiency with hierarchical filter groups​​
• 2017-CVPR-​​All You Need is Beyond a Good Init: Exploring Better Solution for Training Extremely Deep Convolutional Neural Networks with Orthonormality and Modulation​​
• 2017-CVPR-ResNeXt-​​Aggregated Residual Transformations for Deep Neural Networks​​
• 2017-CVPR-​​Xception: Deep learning with depthwise separable convolutions​​
• 2017-ICCV-​​Channel pruning for accelerating very deep neural networks​​​ [​​Code​​]
• 2017-ICCV-​​Learning efficient convolutional networks through network slimming​​​ [​​Code​​]
• 2017-ICCV-​​ThiNet: A filter level pruning method for deep neural network compression​​​ [​​Project​​]
• 2017-ICCV-​​Interleaved group convolutions​​
• 2017-NIPS-​​Net-trim: Convex pruning of deep neural networks with performance guarantee​​
• 2017-NIPS-​​Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon​​
• 2017-NNs-​​Nonredundant sparse feature extraction using autoencoders with receptive fields clustering​​
• 2017.02-​​The Power of Sparsity in Convolutional Neural Networks​​
• 2018-AAAI-​​Auto-balanced Filter Pruning for Efficient Convolutional Neural Networks​​
• 2018-AAAI-​​Deep Neural Network Compression with Single and Multiple Level Quantization​​
• 2018-ICML-​​On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization​​
• 2018-ICMLw-​​Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures​​
• 2018-ICLRo-​​Training and Inference with Integers in Deep Neural Networks​​
• 2018-ICLR-​​Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers​​
• 2018-ICLR-​​N2N learning: Network to Network Compression via Policy Gradient Reinforcement Learning​​
• 2018-ICLR-​​Model compression via distillation and quantization​​
• 2018-ICLR-​​Towards Image Understanding from Deep Compression Without Decoding​​
• 2018-ICLR-​​Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training​​
• 2018-ICLR-​​Mixed Precision Training of Convolutional Neural Networks using Integer Operations​​
• 2018-ICLR-​​Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy​​
• 2018-ICLR-​​Loss-aware Weight Quantization of Deep Networks​​
• 2018-ICLR-​​Alternating Multi-bit Quantization for Recurrent Neural Networks​​
• 2018-ICLR-​​Adaptive Quantization of Neural Networks​​
• 2018-ICLR-​​Variational Network Quantization​​
• 2018-ICLR-​​Learning Sparse Neural Networks through L0 Regularization​​
• 2018-ICLRw-​​To Prune, or Not to Prune: Exploring the Efficacy of Pruning for Model Compression​​​ (Similar topic:​​2018-NIPSw-nip in the bud​​​,​​2018-NIPSw-rethink​​)
• 2018-ICLRw-​​Systematic Weight Pruning of DNNs using Alternating Direction Method of Multipliers​​
• 2018-ICLRw-​​Weightless: Lossy weight encoding for deep neural network compression​​
• 2018-ICLRw-​​Variance-based Gradient Compression for Efficient Distributed Deep Learning​​
• 2018-ICLRw-​​Stacked Filters Stationary Flow For Hardware-Oriented Acceleration Of Deep Convolutional Neural Networks​​
• 2018-ICLRw-​​Training Shallow and Thin Networks for Acceleration via Knowledge Distillation with Conditional Adversarial Networks​​
• 2018-ICLRw-​​Accelerating Neural Architecture Search using Performance Prediction​​
• 2018-ICLRw-​​Nonlinear Acceleration of CNNs​​
• 2018-CVPR-​​Context-Aware Deep Feature Compression for High-Speed Visual Tracking​​
• 2018-CVPR-​​NISP: Pruning Networks using Neuron Importance Score Propagation​​
• 2018-CVPR-​​“Learning-Compression” Algorithms for Neural Net Pruning​​
• 2018-CVPR-​​Deep Image Prior​​​ [​​Code​​]
• 2018-CVPR-​​Condensenet: An efficient densenet using learned group convolutions​​
• 2018-CVPR-​​Shift: A zero flop, zero parameter alternative to spatial convolutions​​
• 2018-CVPR-​​Interleaved structured sparse convolutional neural networks​​
• 2018-IJCAI-​​Efficient DNN Neuron Pruning by Minimizing Layer-wise Nonlinear Reconstruction Error​​
• 2018-IJCAI-​​Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks​​
• 2018-IJCAI-​​Where to Prune: Using LSTM to Guide End-to-end Pruning​​
• 2018-IJCAI-​​Accelerating Convolutional Networks via Global & Dynamic Filter Pruning​​
• 2018-IJCAI-​​Optimization based Layer-wise Magnitude-based Pruning for DNN Compression​​
• 2018-IJCAI-​​Progressive Blockwise Knowledge Distillation for Neural Network Acceleration​​
• 2018-IJCAI-​​Complementary Binary Quantization for Joint Multiple Indexing​​
• 2018-ECCV-​​A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers​​
• 2018-ECCV-​​Coreset-Based Neural Network Compression​​
• 2018-ECCV-​​Data-Driven Sparse Structure Selection for Deep Neural Networks​​​ [​​Code​​]
• 2018-BMVCo-​​Structured Probabilistic Pruning for Convolutional Neural Network Acceleration​​
• 2018-BMVC-​​Efficient Progressive Neural Architecture Search​​
• 2018-BMVC-​​Igcv3: Interleaved lowrank group convolutions for efficient deep neural networks​​
• 2018-NIPS-​​Discrimination-aware Channel Pruning for Deep Neural Networks​​
• 2018-NIPS-​​Frequency-Domain Dynamic Pruning for Convolutional Neural Networks​​
• 2018-NIPS-​​ChannelNets: Compact and Efficient Convolutional Neural Networks via Channel-Wise Convolutions​​
• 2018-NIPS-​​DropBlock: A regularization method for convolutional networks​​
• 2018-NIPS-​​Constructing fast network through deconstruction of convolution​​
• 2018-NIPS-​​Learning Versatile Filters for Efficient Convolutional Neural Networks​​​ [​​Code​​]
• 2018-NIPSw-​​Pruning neural networks: is it time to nip it in the bud?​​
• 2018-NIPSwb-​​Rethinking the Value of Network Pruning​​​ [​​2019 ICLR version​​]
• 2018-NIPSw-​​Structured Pruning for Efficient ConvNets via Incremental Regularization​​
• 2018.05-​​Compression of Deep Convolutional Neural Networks under Joint Sparsity Constraints​​
• 2018.05-​​AutoPruner: An End-to-End Trainable Filter Pruning Method for Efficient Deep Model Inference​​
• 2018.11-​​Second-order Optimization Method for Large Mini-batch: Training ResNet-50 on ImageNet in 35 Epochs​​
• 2018.11-​​Rethinking ImageNet Pre-training​​ (Kaiming He)
• 2019-ICLRo-​​The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks​​
• 2019-AAAIo-​​A layer decomposition-recomposition framework for neuron pruning towards accurate lightweight networks​​
• 2019-CVPR-​​All You Need is a Few Shifts: Designing Efficient Convolutional Neural Networks for Image Classification​​
• 2019-CVPR-​​HetConv Heterogeneous Kernel-Based Convolutions for Deep CNNs​​
• 2019-CVPR-​​Fully Learnable Group Convolution for Acceleration of Deep Neural Networks​​
• 2019-CVPR-​​Towards Optimal Structured CNN Pruning via Generative Adversarial Learning​​
• 2019-CVPR-​​Centripetal SGD for Pruning Very Deep Convolutional Networks with Complicated Structure​​
• 2019-BigComp-​​Towards Robust Compressed Convolutional Neural Networks​​
• 2019-PR-​​Filter-in-Filter: Improve CNNs in a Low-cost Way by Sharing Parameters among the Sub-filters of a Filter​​
• 2019-PRL-​​BDNN: Binary Convolution Neural Networks for Fast Object Detection​​
• 2019.03-​​MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning​​ (Face++)
• 2019.04-​​Data-Free Learning of Student Networks​​ (Huawei)
• 2019.04-​​Resource Efficient 3D Convolutional Neural Networks​​
• 2019.04-​​Meta Filter Pruning to Accelerate Deep Convolutional Neural Networks​​
• 2019.04-​​Knowledge Squeezed Adversarial Network Compression​​​- 2011-JMLR-​​Learning with Structured Sparsity​​
• 2013-NIPS-​​Predicting Parameters in Deep Learning​​
• 2014-BMVC-​​Speeding up convolutional neural networks with low rank expansions​​
• 2014-NIPS-​​Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation​​
• 2014-NIPS-​​Do deep neural nets really need to be deep​​
• 2015-ICML-​​Compressing neural networks with the hashing trick​​
• 2015-INTERSPEECH-​​A Diversity-Penalizing Ensemble Training Method for Deep Learning​​
• 2015-BMVC-​​Data-free parameter pruning for deep neural networks​​
• 2015-NIPS-​​Learning both Weights and Connections for Efficient Neural Network​​
• 2015_NIPSw-​​Distilling Intractable Generative Models​​
• 2015-CVPR-​​Learning to generate chairs with convolutional neural networks​​
• 2015-CVPR-​​Understanding deep image representations by inverting them​​​ [2016 IJCV version:​​Visualizing deep convolutional neural networks using natural pre-images​​]
• 2015-CVPR-​​Efficient and Accurate Approximations of Nonlinear Convolutional Networks​​​ [2016 TPAMI version:​​Accelerating Very Deep Convolutional Networks for Classification and Detection​​]
• 2016-ICLRb-​​Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding​​
• 2016-ICLR-​​All you need is a good init​​​ [​​Code​​]
• 2016-ICLR-​​Convolutional neural networks with low-rank regularization​​
• 2016-ICLR-​​Diversity networks​​
• 2016-EMNLP-​​Sequence-Level Knowledge Distillation​​
• 2016-CVPR-​​Inverting Visual Representations with Convolutional Networks​​
• 2016-NIPS-​​Learning Structured Sparsity in Deep Neural Networks​​
• 2016-NIPS-​​Dynamic Network Surgery for Efficient DNNs​​
• 2016.10-​​Deep model compression: Distilling knowledge from noisy teachers​​
• 2017-ICLR-​​Pruning Convolutional Neural Networks for Resource Efficient Inference​​
• 2017-ICLR-​​Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights​​
• 2017-ICLR-​​Do Deep Convolutional Nets Really Need to be Deep and Convolutional?​​
• 2017-ICML-​​Variational dropout sparsifies deep neural networks​​
• 2017-CVPR-​​Learning deep CNN denoiser prior for image restoration​​
• 2017-CVPR-​​Deep roots: Improving cnn efficiency with hierarchical filter groups​​
• 2017-CVPR-​​All You Need is Beyond a Good Init: Exploring Better Solution for Training Extremely Deep Convolutional Neural Networks with Orthonormality and Modulation​​
• 2017-CVPR-ResNeXt-​​Aggregated Residual Transformations for Deep Neural Networks​​
• 2017-CVPR-​​Xception: Deep learning with depthwise separable convolutions​​
• 2017-ICCV-​​Channel pruning for accelerating very deep neural networks​​​ [​​Code​​]
• 2017-ICCV-​​Learning efficient convolutional networks through network slimming​​​ [​​Code​​]
• 2017-ICCV-​​ThiNet: A filter level pruning method for deep neural network compression​​​ [​​Project​​]
• 2017-ICCV-​​Interleaved group convolutions​​
• 2017-NIPS-​​Net-trim: Convex pruning of deep neural networks with performance guarantee​​
• 2017-NIPS-​​Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon​​
• 2017-NNs-​​Nonredundant sparse feature extraction using autoencoders with receptive fields clustering​​
• 2017.02-​​The Power of Sparsity in Convolutional Neural Networks​​
• 2018-AAAI-​​Auto-balanced Filter Pruning for Efficient Convolutional Neural Networks​​
• 2018-AAAI-​​Deep Neural Network Compression with Single and Multiple Level Quantization​​
• 2018-ICML-​​On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization​​
• 2018-ICMLw-​​Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures​​
• 2018-ICLRo-​​Training and Inference with Integers in Deep Neural Networks​​
• 2018-ICLR-​​Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers​​
• 2018-ICLR-​​N2N learning: Network to Network Compression via Policy Gradient Reinforcement Learning​​
• 2018-ICLR-​​Model compression via distillation and quantization​​
• 2018-ICLR-​​Towards Image Understanding from Deep Compression Without Decoding​​
• 2018-ICLR-​​Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training​​
• 2018-ICLR-​​Mixed Precision Training of Convolutional Neural Networks using Integer Operations​​
• 2018-ICLR-​​Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy​​
• 2018-ICLR-​​Loss-aware Weight Quantization of Deep Networks​​
• 2018-ICLR-​​Alternating Multi-bit Quantization for Recurrent Neural Networks​​
• 2018-ICLR-​​Adaptive Quantization of Neural Networks​​
• 2018-ICLR-​​Variational Network Quantization​​
• 2018-ICLR-​​Learning Sparse Neural Networks through L0 Regularization​​
• 2018-ICLRw-​​To Prune, or Not to Prune: Exploring the Efficacy of Pruning for Model Compression​​​ (Similar topic:​​2018-NIPSw-nip in the bud​​​,​​2018-NIPSw-rethink​​)
• 2018-ICLRw-​​Systematic Weight Pruning of DNNs using Alternating Direction Method of Multipliers​​
• 2018-ICLRw-​​Weightless: Lossy weight encoding for deep neural network compression​​
• 2018-ICLRw-​​Variance-based Gradient Compression for Efficient Distributed Deep Learning​​
• 2018-ICLRw-​​Stacked Filters Stationary Flow For Hardware-Oriented Acceleration Of Deep Convolutional Neural Networks​​
• 2018-ICLRw-​​Training Shallow and Thin Networks for Acceleration via Knowledge Distillation with Conditional Adversarial Networks​​
• 2018-ICLRw-​​Accelerating Neural Architecture Search using Performance Prediction​​
• 2018-ICLRw-​​Nonlinear Acceleration of CNNs​​
• 2018-CVPR-​​Context-Aware Deep Feature Compression for High-Speed Visual Tracking​​
• 2018-CVPR-​​NISP: Pruning Networks using Neuron Importance Score Propagation​​
• 2018-CVPR-​​“Learning-Compression” Algorithms for Neural Net Pruning​​
• 2018-CVPR-​​Deep Image Prior​​​ [​​Code​​]
• 2018-CVPR-​​Condensenet: An efficient densenet using learned group convolutions​​
• 2018-CVPR-​​Shift: A zero flop, zero parameter alternative to spatial convolutions​​
• 2018-CVPR-​​Interleaved structured sparse convolutional neural networks​​
• 2018-IJCAI-​​Efficient DNN Neuron Pruning by Minimizing Layer-wise Nonlinear Reconstruction Error​​
• 2018-IJCAI-​​Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks​​
• 2018-IJCAI-​​Where to Prune: Using LSTM to Guide End-to-end Pruning​​
• 2018-IJCAI-​​Accelerating Convolutional Networks via Global & Dynamic Filter Pruning​​
• 2018-IJCAI-​​Optimization based Layer-wise Magnitude-based Pruning for DNN Compression​​
• 2018-IJCAI-​​Progressive Blockwise Knowledge Distillation for Neural Network Acceleration​​
• 2018-IJCAI-​​Complementary Binary Quantization for Joint Multiple Indexing​​
• 2018-ECCV-​​A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers​​
• 2018-ECCV-​​Coreset-Based Neural Network Compression​​
• 2018-ECCV-​​Data-Driven Sparse Structure Selection for Deep Neural Networks​​​ [​​Code​​]
• 2018-BMVCo-​​Structured Probabilistic Pruning for Convolutional Neural Network Acceleration​​
• 2018-BMVC-​​Efficient Progressive Neural Architecture Search​​
• 2018-BMVC-​​Igcv3: Interleaved lowrank group convolutions for efficient deep neural networks​​
• 2018-NIPS-​​Discrimination-aware Channel Pruning for Deep Neural Networks​​
• 2018-NIPS-​​Frequency-Domain Dynamic Pruning for Convolutional Neural Networks​​
• 2018-NIPS-​​ChannelNets: Compact and Efficient Convolutional Neural Networks via Channel-Wise Convolutions​​
• 2018-NIPS-​​DropBlock: A regularization method for convolutional networks​​
• 2018-NIPS-​​Constructing fast network through deconstruction of convolution​​
• 2018-NIPS-​​Learning Versatile Filters for Efficient Convolutional Neural Networks​​​ [​​Code​​]
• 2018-NIPSw-​​Pruning neural networks: is it time to nip it in the bud?​​
• 2018-NIPSwb-​​Rethinking the Value of Network Pruning​​​ [​​2019 ICLR version​​]
• 2018-NIPSw-​​Structured Pruning for Efficient ConvNets via Incremental Regularization​​
• 2018.05-​​Compression of Deep Convolutional Neural Networks under Joint Sparsity Constraints​​
• 2018.05-​​AutoPruner: An End-to-End Trainable Filter Pruning Method for Efficient Deep Model Inference​​
• 2018.11-​​Second-order Optimization Method for Large Mini-batch: Training ResNet-50 on ImageNet in 35 Epochs​​
• 2018.11-​​Rethinking ImageNet Pre-training​​ (Kaiming He)
• 2019-ICLRo-​​The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks​​
• 2019-AAAIo-​​A layer decomposition-recomposition framework for neuron pruning towards accurate lightweight networks​​
• 2019-CVPR-​​All You Need is a Few Shifts: Designing Efficient Convolutional Neural Networks for Image Classification​​
• 2019-CVPR-​​HetConv Heterogeneous Kernel-Based Convolutions for Deep CNNs​​
• 2019-CVPR-​​Fully Learnable Group Convolution for Acceleration of Deep Neural Networks​​
• 2019-CVPR-​​Towards Optimal Structured CNN Pruning via Generative Adversarial Learning​​
• 2019-CVPR-​​Centripetal SGD for Pruning Very Deep Convolutional Networks with Complicated Structure​​
• 2019-BigComp-​​Towards Robust Compressed Convolutional Neural Networks​​
• 2019-PR-​​Filter-in-Filter: Improve CNNs in a Low-cost Way by Sharing Parameters among the Sub-filters of a Filter​​
• 2019-PRL-​​BDNN: Binary Convolution Neural Networks for Fast Object Detection​​
• 2019.03-​​MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning​​ (Face++)
• 2019.04-​​Data-Free Learning of Student Networks​​ (Huawei)
• 2019.04-​​Resource Efficient 3D Convolutional Neural Networks​​
• 2019.04-​​Meta Filter Pruning to Accelerate Deep Convolutional Neural Networks​​
• 2019.04-​​Knowledge Squeezed Adversarial Network Compression​​