................................................................................................................................... iv ACKNOWLEDGEMENTS ............................................................................................................ vi TABLE OF CONTENTS ............................................................................................................... vii LIST OF TABLES .......................................................................................................................... ix LIST OF FIGURES ......................................................................................................................... x CHAPTER 1: INRTODUCTION .................................................................................................... 1 1.1 Research Problem and Motivation ......................................................................................... 7 1.2 Contributions ......................................................................................................................... 9 CHAPTER 2: LITERATURE SURVEY ....................................................................................... 11 2.1 Traditional Methods for Natural Language Processing ....................................................... 11 2.1.1 N-gram Models ............................................................................................................. 13 2.1.2 Structured Language Models ........................................................................................ 14 2.1.3 Word Vector Representations ....................................................................................... 15 2.2 Neural Networks: Basics and Definitions ............................................................................ 16 2.2.1 Neural Network Language Models (NNLMs) .............................................................. 19 2.2.2 Feedforward Neural Network Based Language Models (FFNNLMs).......................... 20 2.3 Deep Learning Background ................................................................................................. 21 2.4 Deep Learning for Natural Language Processing ................................................................ 22 2.4.1 Windows-Based Neural Networks ................................................................................ 24 2.5 Convolutional Neural Networks (CNNs) ............................................................................. 26 2.5.1 Pooling Layer ................................................................................................................ 27 2.6 Convolution Neural Networks for Natural Language Processing (CNNs-NLP) ................. 29 2.7 GoogLeNet: Inception Convolution Neural Networks ........................................................ 31 2.8 Recurrent Neural Networks (RNNs) .................................................................................... 35 2.8.1 Recurrent Neural Networks Based Language Models (RNNLMs) .............................. 39 2.8.2 The Problem of Long-Term Dependencies ................................................................... 40 2.8.3 Vanishing and Exploding Gradients ............................................................................. 42 2.9 Long Short-Term Memory (LSTM) .................................................................................... 43 2.10 Bidirectional Recurrent Neural Networks (BRNNs) ......................................................... 45 2.11 Gated Recurrent Unite (GRU) ........................................................................................... 46 2.12 Vector Representations of Words ...................................................................................... 47 2.13 Combination of Convolution Neural Networks and Recurrent Neural Networks (CNNsRNNs) ........................................................................................................................................ 49 CHAPTER 3: RESEARCH PLAN ................................................................................................ 51 3.1 Deep Neural Network Language Model for Text Classification ......................................... 51 3.2 The Embedding Layer .......................................................................................................... 51