Scaling recurrent neural network language models

@article{Williams2015ScalingRN,
  title={Scaling recurrent neural network language models},
  author={Will Williams and Niranjani Prasad and David Mrva and Tom Ash and Tony Robinson},
  journal={2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  year={2015},
  pages={5391-5395}
}
This paper investigates the scaling properties of Recurrent Neural Network Language Models (RNNLMs). We discuss how to train very large RNNs on GPUs and address the questions of how RNNLMs scale with respect to model size, training-set size, computational costs and memory. Our analysis shows that despite being more costly to train, RNNLMs obtain much lower perplexities on standard benchmarks than n-gram models. We train the largest known RNNs and present relative word error rates gains of 18… CONTINUE READING
Highly Cited
This paper has 60 citations. REVIEW CITATIONS
Recent Discussions
This paper has been referenced on Twitter 25 times over the past 90 days. VIEW TWEETS

From This Paper

Figures, tables, results, and topics from this paper.

Key Quantitative Results

  • We train the largest known RNNs and present relative word error rates gains of 18% on an ASR task. We also present the new lowest perplexities on the recently released billion word language modelling benchmark, 1 BLEU point gain on machine translation and a 17% relative hit rate gain in word prediction.

Citations

Publications citing this paper.
Showing 1-10 of 39 extracted citations

61 Citations

010203020142015201620172018
Citations per Year
Semantic Scholar estimates that this publication has 61 citations based on the available data.

See our FAQ for additional information.

References

Publications referenced by this paper.
Showing 1-10 of 23 references

Similar Papers

Loading similar papers…