Bimodal Distributed Binarized Neural Networks

@article{Rozen2022BimodalDB,
  title={Bimodal Distributed Binarized Neural Networks},
  author={Tal Rozen and Moshe Kimhi and Brian Chmiel and Avi Mendelson and Chaim Baskin},
  journal={ArXiv},
  year={2022},
  volume={abs/2204.02004}
}
Binary neural networks (BNNs) are an extremely promising method for reducing deep neural networks’ complexity and power consumption significantly. Binarization techniques, however, suffer from ineligible performance degradation compared to their full-precision counterparts. Prior work mainly focused on strategies for sign function approximation during the forward and backward phases to reduce the quantization error during the binarization process. In this work, we propose a bimodal-distributed… 

Figures and Tables from this paper

References

SHOWING 1-10 OF 35 REFERENCES

ImageNet: A large-scale hierarchical image database

A new database called “ImageNet” is introduced, a large-scale ontology of images built upon the backbone of the WordNet structure, much larger in scale and diversity and much more accurate than the current image datasets.

ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions

This paper proposes to generalize the traditional Sign and PReLU functions to enable explicit learning of the distribution reshape and shift at near-zero extra cost and shows that the proposed ReActNet outperforms all the state-of-the-arts by a large margin.

XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks

The Binary-Weight-Network version of AlexNet is compared with recent network binarization methods, BinaryConnect and BinaryNets, and outperform these methods by large margins on ImageNet, more than \(16\,\%\) in top-1 accuracy.

Deep Residual Learning for Image Recognition

This work presents a residual learning framework to ease the training of networks that are substantially deeper than those used previously, and provides comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth.

Post training 4-bit quantization of convolutional networks for rapid-deployment

This paper introduces the first practical 4-bit post training quantization approach: it does not involve training the quantized model (fine-tuning), nor it requires the availability of the full dataset, and achieves accuracy that is just a few percents less the state-of-the-art baseline across a wide range of convolutional models.

Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithm

A novel model, dubbed Bi-Real net, which connects the real activations (after the 1-bit convolution and/or BatchNorm layer, before the sign function) to activations of the consecutive block, through an identity shortcut is proposed, which achieves up to 10% higher top-1 accuracy with more memory saving and lower computational cost.

Distilling the Knowledge in a Neural Network

This work shows that it can significantly improve the acoustic model of a heavily used commercial system by distilling the knowledge in an ensemble of models into a single model and introduces a new type of ensemble composed of one or more full models and many specialist models which learn to distinguish fine-grained classes that the full models confuse.

Learning Multiple Layers of Features from Tiny Images

It is shown how to train a multi-layer generative model that learns to extract meaningful features which resemble those found in the human visual cortex, using a novel parallelization algorithm to distribute the work among multiple machines connected on a network.

A ConvNet for the 2020s

This work gradually “modernize” a standard ResNet toward the design of a vision Transformer, and discovers several key components that contribute to the performance difference along the way, leading to a family of pure ConvNet models dubbed ConvNeXt.

Self-distribution binary neural networks

Self-Distribution Binary Neural Network (SD-BNN) is proposed, which utilizes Activation Self Distribution to adaptively adjust the sign distribution of activations, thereby improving the sign differences of the outputs of the convolution.