Just Add Functions: A Neural-Symbolic Language Model

@inproceedings{Demeter2020JustAF,
  title={Just Add Functions: A Neural-Symbolic Language Model},
  author={David Demeter and Doug Downey},
  booktitle={AAAI},
  year={2020}
}
Neural network language models (NNLMs) have achieved ever-improving accuracy due to more sophisticated architectures and increasing amounts of training data. However, the inductive bias of these models (formed by the distributional hypothesis of language), while ideally suited to modeling most running text, results in key limitations for today's models. In particular, the models often struggle to learn certain spatial, temporal, or quantitative relationships, which are commonplace in text and… 
View PDF on arXiv
4 Citations
Background Citations
1
Methods Citations
1

Figures and Tables from this paper

4 Citations

Understanding in Artificial Intelligence
TLDR
How progress has been made in benchmark development to measure understanding capabilities of AI methods is shown and as well how current methods develop understanding capabilities are reviewed.
Neuro-Symbolic Artificial Intelligence: Current Trends
TLDR
A structured overview of current trends in neuro-Symbolic Artificial Intelligence is provided, by means of categorizing recent publications from key conferences, to serve as a convenient starting point for research on the general topic.
Neuro-symbolic artificial intelligence
TLDR
A structured overview of current trends in neuro-Symbolic Artificial Intelligence is provided, by means of categorizing recent publications from key conferences, to serve as a convenient starting point for research on the general topic.
Is Neuro-Symbolic AI Meeting its Promise in Natural Language Processing? A Structured Review
TLDR
A structured review of studies implementing NeSy for NLP is conducted, with the aim of answering the question of whether NeSy is indeed meeting its promises: reasoning, out-of-distribution generalization, interpretability, learning and reasoning from small data, and transferability to new domains.

References

SHOWING 1-10 OF 25 REFERENCES
A Neural Knowledge Language Model
TLDR
A Neural Knowledge Language Model (NKLM) which combines symbolic knowledge provided by a knowledge graph with the RNN language model, and shows that the NKLM significantly improves the perplexity while generating a much smaller number of unknown words.
Neural Probabilistic Language Models
TLDR
This work proposes to fight the curse of dimensionality by learning a distributed representation for words which allows each training sentence to inform the model about an exponential number of semantically neighboring sentences, and incorporates this new language model into a state-of-the-art speech recognizer of conversational speech.
Exploring the Limits of Language Modeling
TLDR
This work explores recent advances in Recurrent Neural Networks for large scale Language Modeling, and extends current models to deal with two key challenges present in this task: corpora and vocabulary sizes, and complex, long term structure of language.
One billion word benchmark for measuring progress in statistical language modeling
TLDR
A new benchmark corpus to be used for measuring progress in statistical language modeling, with almost one billion words of training data, is proposed, which is useful to quickly evaluate novel language modeling techniques, and to compare their contribution when combined with other advanced techniques.
GloVe: Global Vectors for Word Representation
TLDR
A new global logbilinear regression model that combines the advantages of the two major model families in the literature: global matrix factorization and local context window methods and produces a vector space with meaningful substructure.
Language Models are Unsupervised Multitask Learners
TLDR
It is demonstrated that language models begin to learn these tasks without any explicit supervision when trained on a new dataset of millions of webpages called WebText, suggesting a promising path towards building language processing systems which learn to perform tasks from their naturally occurring demonstrations.
On Using Very Large Target Vocabulary for Neural Machine Translation
TLDR
It is shown that decoding can be efficiently done even with the model having a very large target vocabulary by selecting only a small subset of the whole target vocabulary.
Numeracy for Language Models: Evaluating and Improving their Ability to Predict Numbers
TLDR
This paper explores different strategies for modelling numerals with language models, such as memorisation and digit-by-digit composition, and proposes a novel neural architecture that uses a continuous probability density function to model numerals from an open vocabulary.
Breaking the Softmax Bottleneck: A High-Rank RNN Language Model
TLDR
It is shown that the expressiveness of Softmax-based models (including the majority of neural language models) is limited by a Softmax bottleneck, and a simple and effective method is proposed to address this issue.
Improving Neural Language Models with a Continuous Cache
TLDR
A simplified version of memory augmented networks, which stores past hidden activations as memory and accesses them through a dot product with the current hidden activation, which is very efficient and scales to very large memory sizes.
...
...