Compressing deep quaternion neural networks with targeted regularization

@article{Vecchi2020CompressingDQ,
  title={Compressing deep quaternion neural networks with targeted regularization},
  author={Riccardo Vecchi and Simone Scardapane and Danilo Comminiello and Aurelio Uncini},
  journal={CAAI Trans. Intell. Technol.},
  year={2020},
  volume={5},
  pages={172-176}
}
In recent years, hyper-complex deep networks (such as complex-valued and quaternion-valued neural networks) have received a renewed interest in the literature. They find applications in multiple fields, ranging from image reconstruction to 3D audio processing. Similar to their real-valued counterparts, quaternion neural networks (QVNNs) require custom regularization strategies to avoid overfitting. In addition, for many real-world applications and embedded implementations, there is the need of… 

Figures and Tables from this paper

A Quaternion-Valued Variational Autoencoder
TLDR
A novel VAE defined in the quaternion domain is proposed, which exploits the properties of quaternION algebra to improve performance while significantly reducing the number of parameters required by the network.
Bayesian Sparsification Methods for Deep Complex-valued Networks
TLDR
The proposed Bayesian technique is verified by conducting a large numerical study of the performance-compression trade-off of C-valued networks on two tasks: image recognition on MNIST-like and CIFAR10 datasets and music transcription on MusicNet.
Bayesian Sparsification of Deep C-valued Networks
TLDR
Sparse Variational Dropout is extended to complex-valued neural networks and the proposed Bayesian technique is verified by conducting a large numerical study of the performancecompression trade-off of C-valued networks on two tasks.
Quaternion Generative Adversarial Networks
TLDR
QGANs exploit the properties of quaternion algebra to process channels as a single entity and capture internal latent relations, while reducing by a factor of 4 the overall number of parameters, and are able to obtain better FID scores than real-valued GANs and to generate visually pleasing images.
An Information-Theoretic Perspective on Proper Quaternion Variational Autoencoders
TLDR
This paper analyze the QVAE under an information-theoretic perspective, studying the ability of the H-proper model to approximate improper distributions as well as the built-in H- Proper ones and the loss of entropy due to the improperness of the input signal.
L G ] 2 2 A pr 2 02 1 A QUATERNION-VALUED VARIATIONAL AUTOENCODER
TLDR
A novel VAE defined in the quaternion domain is proposed, which exploits the properties of quaternION algebra to improve performance while significantly reducing the number of parameters required by the network.
AlgebraNets
TLDR
This work considers a richer set of objects for activations and weights, and undertakes a comprehensive study of alternative algebras as number representations by studying their performance on two challenging problems: large-scale image classification using the ImageNet dataset and language modeling using the enwiki8 and WikiText-103 datasets.
Hypercomplex Image-to-Image Translation
TLDR
This paper proposes to leverage hypercomplex algebra properties to leverage lightweight I2I generative models capable of preserving pre-existing relations among images dimensions, thus exploiting additional input information.
Sentiment analysis of students in ideological and political teaching based on artificial intelligence and data mining
TLDR
The research results show that comprehensive analysis of multiple factors can effectively improve the effect of sentiment analysis of students in ideological and political classrooms, and enhance theeffect of ideological andpolitical classroom teaching.
...
...

References

SHOWING 1-10 OF 37 REFERENCES
Quaternion Convolutional Neural Networks
TLDR
This work re-designs the basic modules like convolution layer and fully-connected layer in the quaternion domain, which can be used to establish fully-quaternion convolutional neural networks, compatible with almost all deep learning techniques and can be plugged into traditional CNNs easily.
Quaternion Convolutional Neural Networks for Heterogeneous Image Processing
TLDR
This paper proposes to explore the fundamental reason of the success of QCNN over CNN, by investigating the impact of the Hamilton product on a color image reconstruction task performed from a gray-scale only training.
Complex-Valued Neural Networks With Nonparametric Activation Functions
TLDR
This paper proposes the first fully complex, nonparametric activation function for CVNNs, which is based on a kernel expansion with a fixed dictionary that can be implemented efficiently on vectorized hardware.
Quaternion Recurrent Neural Networks
TLDR
It is shown that both QRNN and QLSTM achieve better performances than RNN and LSTM in a realistic application of automatic speech recognition, and reduce by a maximum factor of 3.3x the number of free parameters needed, compared to real-valued RNNs and L STMs to reach better results.
Learning Structured Sparsity in Deep Neural Networks
TLDR
The results show that for CIFAR-10, regularization on layer depth can reduce 20 layers of a Deep Residual Network to 18 layers while improve the accuracy from 91.25% to 92.60%, which is still slightly higher than that of original ResNet with 32 layers.
Optimization in Quaternion Dynamic Systems: Gradient, Hessian, and Learning Algorithms
TLDR
New definitions of quaternion gradient and Hessian are proposed, based on the novel generalized Hamilton-real (GHR) calculus, thus making a possible efficient derivation of general optimization algorithms directly in the quaternions field, rather than using the isomorphism with the real domain, as is current practice.
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
TLDR
Applied to a state-of-the-art image classification model, Batch Normalization achieves the same accuracy with 14 times fewer training steps, and beats the original model by a significant margin.
Quaternion Convolutional Neural Networks for Detection and Localization of 3D Sound Events
TLDR
This paper considers the spherical harmonic components of the signals captured by a first-order ambisonic microphone and process them by using a quaternion convolutional neural network, thus improving accuracy results in 3D sound event detection and localization.
...
...