Corpus ID: 211069143

Soft Threshold Weight Reparameterization for Learnable Sparsity

@article{Kusupati2020SoftTW,
  title={Soft Threshold Weight Reparameterization for Learnable Sparsity},
  author={Aditya Kusupati and V. Ramanujan and Raghav Somani and Mitchell Wortsman and Prateek Jain and S. Kakade and A. Farhadi},
  journal={ArXiv},
  year={2020},
  volume={abs/2002.03231}
}
Sparsity in Deep Neural Networks (DNNs) is studied extensively with the focus of maximizing prediction accuracy given an overall parameter budget. Existing methods rely on uniform or heuristic non-uniform sparsity budgets which have sub-optimal layer-wise parameter allocation resulting in a) lower prediction accuracy or b) higher inference cost (FLOPs). This work proposes Soft Threshold Reparameterization (STR), a novel use of the soft-threshold operator on DNN weights. STR smoothly induces… Expand
25 Citations
HALO: Learning to Prune Neural Networks with Shrinkage.
  • PDF
A Deeper Look at the Layerwise Sparsity of Magnitude-based Pruning
  • PDF
Learning N: M Fine-grained Structured Sparse Neural Networks From Scratch
  • 1
  • Highly Influenced
  • PDF
Selfish Sparse RNN Training
  • 3
  • PDF
Adapting Unstructured Sparsity Techniques for Structured Sparsity
  • 1
  • PDF
Rigging the Lottery: Making All Tickets Winners
  • 62
  • PDF
Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training
  • 1
  • Highly Influenced
  • PDF
RNNPool: Efficient Non-linear Pooling for RAM Constrained Inference
  • 7
  • PDF
Learnable Embedding Sizes for Recommender Systems
  • Highly Influenced
  • PDF
...
1
2
3
...

References

SHOWING 1-10 OF 65 REFERENCES
Learning Structured Sparsity in Deep Neural Networks
  • 1,192
  • PDF
Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization
  • 65
  • Highly Influential
  • PDF
Data-Driven Sparse Structure Selection for Deep Neural Networks
  • 200
  • PDF
The State of Sparsity in Deep Neural Networks
  • 181
  • Highly Influential
  • PDF
Learned Threshold Pruning
  • 5
  • PDF
SNIP: Single-shot Network Pruning based on Connection Sensitivity
  • 210
  • PDF
To prune, or not to prune: exploring the efficacy of pruning for model compression
  • 332
  • Highly Influential
  • PDF
Winning the Lottery with Continuous Sparsification
  • 16
  • PDF
Rigging the Lottery: Making All Tickets Winners
  • 62
  • PDF
Sparse Convolutional Neural Networks
  • 493
  • PDF
...
1
2
3
4
5
...