FReLU: Flexible Rectified Linear Units for Improving Convolutional Neural Networks
@article{Qiu2017FReLUFR, title={FReLU: Flexible Rectified Linear Units for Improving Convolutional Neural Networks}, author={Suo Qiu and Bolun Cai}, journal={2018 24th International Conference on Pattern Recognition (ICPR)}, year={2017}, pages={1223-1228} }
Rectified linear unit (ReLU) is a widely used activation function for deep convolutional neural networks. However, because of the zero-hard rectification, ReLU networks lose the benefits from negative values. In this paper, we propose a novel activation function called flexible rectified linear unit (FReLU) to further explore the effects of negative values. By redesigning the rectified point of ReLU as a learnable parameter, FReLU expands the states of the activation output. When a network is…
Figures and Tables from this paper
46 Citations
Rectified Exponential Units for Convolutional Neural Networks
- Computer ScienceIEEE Access
- 2019
This paper proposes a novel activation function called Rectified Exponential Unit (REU), inspired by two recently proposed activation functions: Exponential Linear Unit (ELU) and Swish, which is designed by introducing the advantage of flexible exponent and multiplication function form.
DPReLU: Dynamic Parametric Rectified Linear Unit
- Computer ScienceSMA
- 2020
DPReLU is proposed that can control the overall functional shape of ReLU with four learnable parameters and is based on the ideas of the Parametric ReLU (PReLU) and Flexible Re LU (FReLU).
Parametric Flatten-T Swish: An Adaptive Non-linear Activation Function For Deep Learning
- Computer ScienceJournal of Information and Communication Technology
- 2020
The proposed Parametric Flatten-T Swish manifested higher non-linear approximation power during training and thereby improved the predictive performance of the networks.
Flatten-T Swish: a thresholded ReLU-Swish-like activation function for deep learning
- Computer ScienceInternational Journal of Advances in Intelligent Informatics
- 2018
An activation function called Flatten-T Swish (FTS) that leverage the benefit of the negative values is proposed and is evaluated, which improves MNIST classification accuracy and converges twice as fast as ReLU.
Soft-Root-Sign Activation Function
- Computer ScienceArXiv
- 2020
The proposed nonlinearity, namely "Soft-Root-Sign" (SRS), is smooth, non-monotonic, and bounded, making it more compatible with batch normalization (BN) and less sensitive to initialization.
TanhSoft—Dynamic Trainable Activation Functions for Faster Learning and Better Performance
- Computer ScienceIEEE Access
- 2021
This work proposes three novel activation functions with learnable parameters, namely TanhSoft-1, Tanh Soft-2, and Tanh soft-3, which are shown to outperform several well-known activation functions.
Activation functions in deep learning: A comprehensive survey and benchmark
- Computer ScienceNeurocomputing
- 2022
Adaptively Customizing Activation Functions for Various Layers
- Computer ScienceArXiv
- 2021
A novel methodology is proposed to adaptively customize activation functions only by adding very few parameters to the traditional activation functions such as Sigmoid, Tanh, and ReLU, and it can surpass other popular methods like ReLU and adaptive functions like Swish in terms of overall performance.
A Comprehensive Survey and Performance Analysis of Activation Functions in Deep Learning
- Computer ScienceArXiv
- 2021
A comprehensive overview and survey is presented for AFs in neural networks for deep learning, covering different classes of AFs such as Logistic Sigmoid and Tanh based, ReLU based, ELU based, and Learning based.
Tensor-based framework for training flexible neural networks
- Computer ScienceArXiv
- 2021
A new learning algorithm is proposed which solves a constrained coupled matrix-tensor factorization (CMTF) problem and fuses the first and zeroth order information of the NN, where the first-order information is contained in a Jacobian tensor following a constrained canonical polyadic decomposition (CPD).
References
SHOWING 1-10 OF 26 REFERENCES
Empirical Evaluation of Rectified Activations in Convolutional Network
- Computer ScienceArXiv
- 2015
The experiments suggest that incorporating a non-zero slope for negative part in rectified activation units could consistently improve the results, and are negative on the common belief that sparsity is the key of good performance in ReLU.
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
- Computer Science2015 IEEE International Conference on Computer Vision (ICCV)
- 2015
This work proposes a Parametric Rectified Linear Unit (PReLU) that generalizes the traditional rectified unit and derives a robust initialization method that particularly considers the rectifier nonlinearities.
Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
- Computer ScienceICLR
- 2016
The "exponential linear unit" (ELU) which speeds up learning in deep neural networks and leads to higher classification accuracies and significantly better generalization performance than ReLUs and LReLUs on networks with more than 5 layers.
Parametric Exponential Linear Unit for Deep Convolutional Neural Networks
- Computer Science2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA)
- 2017
The results on the MNIST, CIFAR-10/100 and ImageNet datasets using the NiN, Overfeat, All-CNN and ResNet networks indicate that the proposed Parametric ELU (PELU) has better performances than the non-parametricELU.
P-TELU: Parametric Tan Hyperbolic Linear Unit Activation for Deep Neural Networks
- Computer Science2017 IEEE International Conference on Computer Vision Workshops (ICCVW)
- 2017
Enhanced performance of the proposed activation function is evaluated on CIFAR10 and CI-FAR100 image dataset using two convolutional neural network architectures: KerasNet, a small 6 layer CNN model, and on 76 layer deep ResNet architecture.
Improving Deep Neural Network with Multiple Parametric Exponential Linear Units
- Computer ScienceNeurocomputing
- 2018
Improving Deep Learning by Inverse Square Root Linear Units (ISRLUs)
- Computer ScienceArXiv
- 2017
The “inverse square root linear unit” (ISRLU) is introduced to speed up learning in deep neural networks and a computationally efficient variant called the “ISRU” which can be used for RNNs is suggested which has less computational complexity but still has a similar curve to tanh and sigmoid.
Systematic evaluation of convolution neural network advances on the Imagenet
- Computer ScienceComput. Vis. Image Underst.
- 2017
Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning
- Computer ScienceAAAI
- 2017
Clear empirical evidence that training with residual connections accelerates the training of Inception networks significantly is given and several new streamlined architectures for both residual and non-residual Inception Networks are presented.
Deep Residual Networks with Exponential Linear Unit
- Computer ScienceArXiv
- 2016
This paper proposes to replace the combination of ReLU and Batch Normalization with Exponential Linear Unit (ELU) in Residual Networks, and shows that this not only speeds up the learning behavior in Residine Networks, but also improves the classification performance as the depth increases.