Efficient Multi-Prize Lottery Tickets: Enhanced Accuracy, Training, and Inference Speed

@article{Cheng2022EfficientML,
  title={Efficient Multi-Prize Lottery Tickets: Enhanced Accuracy, Training, and Inference Speed},
  author={Hao Cheng and Pu Zhao and Yize Li and Xue Lin and James Diffenderfer and Ryan A. Goldhahn and Bhavya Kailkhura},
  journal={ArXiv},
  year={2022},
  volume={abs/2209.12839}
}
Recently, Diffenderfer & Kailkhura [1] proposed a new paradigm for learning compact yet highly accurate binary neural networks simply by pruning and quantizing randomly weighted full precision neural networks. However, the accuracy of these multi-prize tickets (MPTs) is highly sensi-tive to the optimal prune ratio, which limits their applicability. Further-more, the original implementation did not attain any training or inference speed benefits. In this report, we discuss several improvements to… 

Figures and Tables from this paper

References

SHOWING 1-8 OF 8 REFERENCES

Multi-Prize Lottery Ticket Hypothesis: Finding Accurate Binary Neural Networks by Pruning A Randomly Weighted Network

Empirical results indicate that as models grow deeper and wider, multi-prize tickets start to reach similar (and sometimes even higher) test accuracy compared to their significantly larger and full-precision counterparts that have been weight-trained.

Powerpropagation: A sparsity inducing weight reparameterisation

This work introduces Powerpropagation, a new weight-parameterisation for neural networks that leads to inherently sparse models and advocates the use of sparsity in overcoming catastrophic forgetting, where compressed representations allow accommodating a large number of tasks at fixed model capacity.

Rethinking the Hyperparameters for Fine-tuning

It is found that picking the right value for momentum is critical for fine-tuning performance and is connected with previous theoretical findings.

What’s Hidden in a Randomly Weighted Neural Network?

It is empirically show that as randomly weighted neural networks with fixed weights grow wider and deeper, an ``untrained subnetwork" approaches a network with learned weights in accuracy.

Unsupervised Finetuning

This paper finds the source data is crucial when shifting the finetuning paradigm from supervise to unsupervise, and proposes two simple and effective strategies to combine source and target data into unsupervised finetuned: “sparse source data replaying”, and “data mixing”.

PCONV: The Missing but Desirable Sparsity in DNN Weight Pruning for Real-time Execution on Mobile Devices

PCONV, comprising a new sparsity dimension, – fine-grained pruning patterns inside the coarse- grained structures, is introduced, comprising two types of sparsities, Sparse Convolution Patterns (SCP) and connectivity sparsity generated from inter-convolution kernel pruning.

Progressive Kernel Pruning Based on the Information Mapping Sparse Index for CNN Compression

Compared to the existing CNN model compression methods, the IMSI-based progressive kernel pruning method exhibits better compression performance in processing the model compression tasks that are currently popular.