PR Product: A Substitute for Inner Product in Neural Networks

@article{Wang2019PRPA,
title={PR Product: A Substitute for Inner Product in Neural Networks},
author={Zhennan Wang and Wenbin Zou and Chen Xu},
journal={2019 IEEE/CVF International Conference on Computer Vision (ICCV)},
year={2019},
pages={6012-6021}
}
• Published 30 April 2019
• Computer Science
• 2019 IEEE/CVF International Conference on Computer Vision (ICCV)
In this paper, we analyze the inner product of weight vector w and data vector x in neural networks from the perspective of vector orthogonal decomposition and prove that the direction gradient of w decreases with the angle between them close to 0 or {\pi}. We propose the Projection and Rejection Product (PR Product) to make the direction gradient of w independent of the angle and consistently larger than the one in standard inner product while keeping the forward propagation identical. [] Key Method As a…
3 Citations
• Computer Science
IEEE Signal Processing Letters
• 2020
A novel discrimination regularization method for image classification, which enhances the intra-class compactness and inter-class discrepancy simultaneously, through decreasing the minimal angle (DMA) between the feature vector and any one of the weight vectors in classification layer.
This work proposes a lightweight network for ASL gesture recognition with a performance sufficient for practical applications and demonstrates impressive robustness on MS-ASL dataset and in live mode for continuous sign gesture recognition scenario.
• Computer Science
2020 13th International Conference on Human System Interaction (HSI)
• 2020
The proposed method introduces Feature Aggregation Module (FAM) and Refinement Module (RM) to obtain more powerful feature pyramids for predicting objects of different scales and integrates the FAM and the RAM into the architecture of Faster R-CNN called EFPN FasterR-CNN.

References

SHOWING 1-10 OF 55 REFERENCES

• Computer Science
ECCV
• 2018
This paper proposes a novel recurrent fusion network (RFNet) for the image captioning task, which can exploit the interactions among the outputs of the image encoders and generate new compact and informative representations for the decoder.
• Computer Science
2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
• 2017
On the ImageNet-1K dataset, it is empirically show that even under the restricted condition of maintaining complexity, increasing cardinality is able to improve classification accuracy and is more effective than going deeper or wider when the authors increase the capacity.
• Computer Science
AISTATS
• 2015
The proposed deeply-supervised nets (DSN) method simultaneously minimizes classification error while making the learning process of hidden layers direct and transparent, and extends techniques from stochastic gradient methods to analyze the algorithm.
• Computer Science
2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
• 2017
The Dense Convolutional Network (DenseNet), which connects each layer to every other layer in a feed-forward fashion, and has several compelling advantages: they alleviate the vanishing-gradient problem, strengthen feature propagation, encourage feature reuse, and substantially reduce the number of parameters.
• Computer Science
2017 IEEE International Conference on Computer Vision (ICCV)
• 2017
This paper presents Long Short-Term Memory with Attributes (LSTM-A) - a novel architecture that integrates attributes into the successful Convolutional Neural Networks plus Recurrent Neural Networks (RNNs) image captioning framework, by training them in an end-to-end manner.
• Kaiming HeJian Sun
• Computer Science
2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
• 2016
This work presents a residual learning framework to ease the training of networks that are substantially deeper than those used previously, and provides comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth.
The propagation formulations behind the residual building blocks suggest that the forward and backward signals can be directly propagated from one block to any other block, when using identity mappings as the skip connections and after-addition activation.
• Computer Science
BMVC
• 2016
This paper conducts a detailed experimental study on the architecture of ResNet blocks and proposes a novel architecture where the depth and width of residual networks are decreased and the resulting network structures are called wide residual networks (WRNs), which are far superior over their commonly used thin and very deep counterparts.
• Computer Science
2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
• 2017
A novel decision-making framework for image captioning that combines a policy network and a value network to collaboratively generate captions and outperforms state-of-the-art approaches across different evaluation metrics.
• Computer Science
ECCV
• 2018
This paper introduces a new design to explore the connections between objects for image captioning under the umbrella of attention-based encoder-decoder framework that novelly integrates both semantic and spatial object relationships into image encoder.