Just Add Functions: A Neural-Symbolic Language Model
@inproceedings{Demeter2020JustAF, title={Just Add Functions: A Neural-Symbolic Language Model}, author={David Demeter and Doug Downey}, booktitle={AAAI}, year={2020} }
Neural network language models (NNLMs) have achieved ever-improving accuracy due to more sophisticated architectures and increasing amounts of training data. However, the inductive bias of these models (formed by the distributional hypothesis of language), while ideally suited to modeling most running text, results in key limitations for today's models. In particular, the models often struggle to learn certain spatial, temporal, or quantitative relationships, which are commonplace in text and…
Figures and Tables from this paper
4 Citations
Understanding in Artificial Intelligence
- Computer ScienceArXiv
- 2021
How progress has been made in benchmark development to measure understanding capabilities of AI methods is shown and as well how current methods develop understanding capabilities are reviewed.
Neuro-Symbolic Artificial Intelligence: Current Trends
- Computer ScienceArXiv
- 2021
A structured overview of current trends in neuro-Symbolic Artificial Intelligence is provided, by means of categorizing recent publications from key conferences, to serve as a convenient starting point for research on the general topic.
Neuro-symbolic artificial intelligence
- Computer ScienceAI Commun.
- 2021
A structured overview of current trends in neuro-Symbolic Artificial Intelligence is provided, by means of categorizing recent publications from key conferences, to serve as a convenient starting point for research on the general topic.
Is Neuro-Symbolic AI Meeting its Promise in Natural Language Processing? A Structured Review
- Computer ScienceArXiv
- 2022
A structured review of studies implementing NeSy for NLP is conducted, with the aim of answering the question of whether NeSy is indeed meeting its promises: reasoning, out-of-distribution generalization, interpretability, learning and reasoning from small data, and transferability to new domains.
References
SHOWING 1-10 OF 25 REFERENCES
A Neural Knowledge Language Model
- Computer ScienceArXiv
- 2016
A Neural Knowledge Language Model (NKLM) which combines symbolic knowledge provided by a knowledge graph with the RNN language model, and shows that the NKLM significantly improves the perplexity while generating a much smaller number of unknown words.
Neural Probabilistic Language Models
- Computer Science
- 2006
This work proposes to fight the curse of dimensionality by learning a distributed representation for words which allows each training sentence to inform the model about an exponential number of semantically neighboring sentences, and incorporates this new language model into a state-of-the-art speech recognizer of conversational speech.
Exploring the Limits of Language Modeling
- Computer ScienceArXiv
- 2016
This work explores recent advances in Recurrent Neural Networks for large scale Language Modeling, and extends current models to deal with two key challenges present in this task: corpora and vocabulary sizes, and complex, long term structure of language.
One billion word benchmark for measuring progress in statistical language modeling
- Computer ScienceINTERSPEECH
- 2014
A new benchmark corpus to be used for measuring progress in statistical language modeling, with almost one billion words of training data, is proposed, which is useful to quickly evaluate novel language modeling techniques, and to compare their contribution when combined with other advanced techniques.
GloVe: Global Vectors for Word Representation
- Computer ScienceEMNLP
- 2014
A new global logbilinear regression model that combines the advantages of the two major model families in the literature: global matrix factorization and local context window methods and produces a vector space with meaningful substructure.
Language Models are Unsupervised Multitask Learners
- Computer Science
- 2019
It is demonstrated that language models begin to learn these tasks without any explicit supervision when trained on a new dataset of millions of webpages called WebText, suggesting a promising path towards building language processing systems which learn to perform tasks from their naturally occurring demonstrations.
On Using Very Large Target Vocabulary for Neural Machine Translation
- Computer ScienceACL
- 2015
It is shown that decoding can be efficiently done even with the model having a very large target vocabulary by selecting only a small subset of the whole target vocabulary.
Numeracy for Language Models: Evaluating and Improving their Ability to Predict Numbers
- Computer ScienceACL
- 2018
This paper explores different strategies for modelling numerals with language models, such as memorisation and digit-by-digit composition, and proposes a novel neural architecture that uses a continuous probability density function to model numerals from an open vocabulary.
Breaking the Softmax Bottleneck: A High-Rank RNN Language Model
- Computer ScienceICLR
- 2018
It is shown that the expressiveness of Softmax-based models (including the majority of neural language models) is limited by a Softmax bottleneck, and a simple and effective method is proposed to address this issue.
Improving Neural Language Models with a Continuous Cache
- Computer ScienceICLR
- 2017
A simplified version of memory augmented networks, which stores past hidden activations as memory and accesses them through a dot product with the current hidden activation, which is very efficient and scales to very large memory sizes.