Skip to search formSkip to main contentSkip to account menu

CORD-19: The COVID-19 Open Research Dataset

TLDR
The mechanics of dataset construction are described, highlighting challenges and key design decisions, an overview of how CORD-19 has been used, and several shared tasks built around the dataset are described.
View on ACL (opens in a new tab)

Sequential Neural Networks as Automata

TLDR
This work first defines what it means for a real-time network with bounded precision to accept a language and defines a measure of network memory, which helps explain neural computation, as well as the relationship between neural networks and natural language grammar.
View on ACL (opens in a new tab)

Provable Limitations of Acquiring Meaning from Ungrounded Form: What Will Future Language Models Understand?

TLDR
This work formalizes ways in which ungrounded language models appear to be fundamentally limited in their ability to “understand”, and suggests that assertions in code or language do not provide sufficient signal to fully emulate semantic representations.
View on MIT Press (opens in a new tab)

Competency Problems: On Finding and Removing Artifacts in Language Data

TLDR
This work argues that for complex language understanding tasks, all simple feature correlations are spurious, and formalizes this notion into a class of problems which are called competency problems, and gives a simple statistical test for dataset artifacts that is used to show more subtle biases.
View on ACL (opens in a new tab)

Context-Free Transductions with Neural Stacks

TLDR
It is shown that stack-augmented RNNs can discover intuitive stack-based strategies for solving tasks, and more complex networks often find approximate solutions by using the stack as unstructured memory.
View on ACL (opens in a new tab)

End-to-End Graph-Based TAG Parsing with Neural Networks

TLDR
This work presents a graph-based Tree Adjoining Grammar (TAG) parser that uses BiLSTMs, highway connections, and character-level CNNs, and demonstrates that the proposed parser achieves state-of-the-art performance in the downstream tasks of Parsing Evaluation using Textual Entailments (PETE) and Unbounded Dependency Recovery.
View on ACL (opens in a new tab)

A Formal Hierarchy of RNN Architectures

TLDR
It is hypothesized that the practical learnable capacity of unsaturated RNNs obeys a similar hierarchy, and empirical results to support this conjecture are provided.
View on ACL (opens in a new tab)

Saturated Transformers are Constant-Depth Threshold Circuits

TLDR
Saturated transformers transcend the known limitations of hard-attention transformers, and it is proved saturated transformers with floating-point values can be simulated by constant-depth threshold circuits, giving the class TC0 as an upper bound on the formal languages they recognize.
View on MIT Press (opens in a new tab)

Formal Language Theory Meets Modern NLP

3 Data structures 5 3.1 Stacks and counters . . . . . . . . . . . . . . . . . . . . . . . . . 7 3.2 Power of unbounded memory . . . . . . . . . . . . . . . . . . . . 8 3.3 Power of counters . . . . .…
View PDF on arXiv (opens in a new tab)

On the Linguistic Capacity of Real-Time Counter Automata

TLDR
This work studies the abilities of real-time counter machines as formal grammars, focusing on formal properties that are relevant for NLP models and makes general contributions to the theory of formal languages that are of potential interest for understanding recurrent neural networks.
View PDF on arXiv (opens in a new tab)
By clicking accept or continuing to use the site, you agree to the terms outlined in our Privacy Policy (opens in a new tab), Terms of Service (opens in a new tab), and Dataset License (opens in a new tab)