• Corpus ID: 35167190

Recurrent Neural Network Structured Output Prediction for Spoken Language Understanding

@inproceedings{Liu2015RecurrentNN,
  title={Recurrent Neural Network Structured Output Prediction for Spoken Language Understanding},
  author={Bing Liu and Ian R. Lane},
  year={2015}
}
Recurrent Neural Networks (RNNs) have been widely used for sequence modeling due to their strong capabilities in modeling temporal dependencies. In this work, we study and evaluate the effectiveness of using RNNs for slot filling, a key task in Spoken Language Understanding (SLU), with special focus on modeling label sequence dependencies. Recent work on slot filling using RNNs did not model label sequence dependencies explicitly. We propose to introduce label dependencies in model training by… 
speech.sv.cmu.edu
53 Citations
Highly Influential Citations
4
Background Citations
13
Methods Citations
29

Figures and Tables from this paper

53 Citations

Citation Type

Citation Type

Joint Online Spoken Language Understanding and Language Modeling With Recurrent Neural Networks
TLDR
A recurrent neural network model that jointly performs intent detection, slot filling, and language modeling is described that outperforms the independent task training model on SLU tasks and shows advantageous performance in the realistic ASR settings with noisy speech input.
Leveraging Sentence-level Information with Encoder LSTM for Natural Language Understanding
TLDR
This paper enhances LSTM-based sequence labeling to explicitly model label dependencies and proposes another enhancement to incorporate the global information spanning over the whole input sequence to predict the label sequence.
ClockWork-RNN Based Architectures for Slot Filling
TLDR
This paper proposes the use of ClockWork recurrent neural network (CW-RNN) architectures in the slot-filling domain and finds that they significantly outperform simple RNNs and they achieve results among the state-of-the-art, while retaining smaller complexity.
Leveraging Sentence-level Information with Encoder LSTM for Semantic Slot Filling
TLDR
This paper enhances LSTM-based sequence labeling to explicitly model label dependencies and proposes another enhancement to incorporate the global information spanning over the whole input sequence to predict the label sequence.
Modeling with Recurrent Neural Networks for Open Vocabulary Slots
TLDR
It is shown that the model outperforms the existing RNN models with respect to discovering ‘open-vocabulary’ slots without any external information, such as a named entity database or knowledge base, and demonstrates superior performance with regard to intent detection.
Attention-Based Recurrent Neural Network Models for Joint Intent Detection and Slot Filling
TLDR
This work proposes an attention-based neural network model for joint intent detection and slot filling, both of which are critical steps for many speech understanding and dialog systems.
A hybrid convolutional and recurrent network approach for conversational AI in spoken language understanding
TLDR
This paper proposes a hybrid architecture, as a combination of a Recurrent Neural Network and a Convolutional Neural Network models, for Slot Filling in Spoken Language Understanding, and demonstrates the effectiveness of hybrid models that combine benefits from both RNN and CNN architecture.
A Joint Learning Framework With BERT for Spoken Language Understanding
TLDR
A novel encoder-decoder framework based multi-task learning model, which conducts joint training for intent classification and slot filling tasks and outperforms the state-of-the-art approaches.
Modeling Token-level Uncertainty to Learn Unknown Concepts in SLU via Calibrated Dirichlet Prior RNN
TLDR
A Dirichlet Prior RNN is designed to model high-order uncertainty by degenerating as softmax layer for RNN model training to enhance the uncertainty modeling robustness and a novel multi-task training to calibrate theDirichlet concentration parameters is proposed.
SASGBC: Improving Sequence Labeling Performance for Joint Learning of Slot Filling and Intent Detection
TLDR
A new joint learning model called SASGBC is proposed that combines sequence labeling with deep bidirectional Transformer and achieves state-of-the-art performance compared to several competitive baseline models.
...
...

References

SHOWING 1-10 OF 23 REFERENCES
Using Recurrent Neural Networks for Slot Filling in Spoken Language Understanding
TLDR
This paper implemented and compared several important RNN architectures, including Elman, Jordan, and hybrid variants, and implemented these networks with the publicly available Theano neural network toolkit and completed experiments on the well-known airline travel information system (ATIS) benchmark.
Recurrent conditional random field for language understanding
TLDR
This paper shows that the performance of an RNN tagger can be significantly improved by incorporating elements of the CRF model; specifically, the explicit modeling of output-label dependencies with transition features, its global sequence-level objective function, and offline decoding.
Investigation of recurrent-neural-network architectures and learning methods for spoken language understanding
TLDR
The results show that on this task, both types of recurrent networks outperform the CRF baseline substantially, and a bi-directional Jordantype network that takes into account both past and future dependencies among slots works best, outperforming a CRFbased baseline by 14% in relative error reduction.
Recurrent neural networks for language understanding
TLDR
This paper modify the architecture to perform Language Understanding, and advance the state-of-the-art for the widely used ATIS dataset.
Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
TLDR
This work proposes a curriculum learning strategy to gently change the training process from a fully guided scheme using the true previous token, towards a less guided scheme which mostly uses the generated token instead.
Sequence to Sequence Learning with Neural Networks
TLDR
This paper presents a general end-to-end approach to sequence learning that makes minimal assumptions on the sequence structure, and finds that reversing the order of the words in all source sentences improved the LSTM's performance markedly, because doing so introduced many short term dependencies between the source and the target sentence which made the optimization problem easier.
Speech recognition with deep recurrent neural networks
TLDR
This paper investigates deep recurrent neural networks, which combine the multiple levels of representation that have proved so effective in deep networks with the flexible use of long range context that empowers RNNs.
Extensions of recurrent neural network language model
TLDR
Several modifications of the original recurrent neural network language model are presented, showing approaches that lead to more than 15 times speedup for both training and testing phases and possibilities how to reduce the amount of parameters in the model.
Bidirectional recurrent neural networks
TLDR
It is shown how the proposed bidirectional structure can be easily modified to allow efficient estimation of the conditional posterior probability of complete symbol sequences without making any explicit assumption about the shape of the distribution.
A unified architecture for natural language processing: deep neural networks with multitask learning
We describe a single convolutional neural network architecture that, given a sentence, outputs a host of language processing predictions: part-of-speech tags, chunks, named entity tags, semantic…
...
...