• Corpus ID: 219531812

Language Modeling for Formal Mathematics

  title={Language Modeling for Formal Mathematics},
  author={Markus N. Rabe and Dennis Lee and Kshitij Bansal and Christian Szegedy},
We examine whether language modeling applied to mathematical formulas enables logical reasoning. We suggest several logical reasoning tasks that can be used to evaluate language models trained on formal mathematical statements, such as type inference, suggesting missing assumptions and completing equalities. To train language models for formal mathematics, we propose a novel skip-tree task, which outperforms standard language modeling tasks on our reasoning benchmarks. We also analyze the… 
1 Citations

Figures and Tables from this paper

Generative Language Modeling for Automated Theorem Proving
This work presents an automated prover and proof assistant, GPT-f, for the Metamath formalization language, and analyzes its performance, finding new short proofs that were accepted into the mainMetamath library, which is to this knowledge, the first time a deep-learning based system has contributed proofs that are adopted by a formal mathematics community.


Learning to Prove Theorems via Interacting with Proof Assistants
ASTactic, a deep learning-based model that generates tactics as programs in the form of abstract syntax trees (ASTs) can generate effective tactics and can be used to prove new theorems not previously provable by automated methods.
Deep Learning for Symbolic Mathematics
It is shown that neural networks can be surprisingly good at more elaborated tasks in mathematics, such as symbolic integration and solving differential equations, and a syntax for representing these mathematical problems, and methods for generating large datasets that can be used to train sequence-to-sequence models.
Language Models are Unsupervised Multitask Learners
It is demonstrated that language models begin to learn these tasks without any explicit supervision when trained on a new dataset of millions of webpages called WebText, suggesting a promising path towards building language processing systems which learn to perform tasks from their naturally occurring demonstrations.
Teaching Temporal Logics to Neural Networks
The Transformer generalized from imperfect training data to the semantics of LTL, and the results were surprising: the Transformer returns the syntactically equivalent trace in 89% of the cases on a held-out test set.
Generating correctness proofs with neural networks
Proverbot9001 is presented, a proof search system using machine learning techniques to produce proofs of software correctness in interactive theorem provers, and exhibits a proof completion rate that is a 4X improvement over prior state-of-the-art machine learning models for generating proofs in Coq.
GamePad: A Learning Environment for Theorem Proving
A system called GamePad is introduced that can be used to explore the application of machine learning methods to theorem proving in the Coq proof assistant and addresses position evaluation and tactic prediction tasks, which arise naturally in tactic-based theorem proving.
HOList: An Environment for Machine Learning of Higher-Order Theorem Proving (extended version)
This work provides an open-source framework based on the HOL Light theorem prover that can be used as a reinforcement learning environment and presents a deep reinforcement learning driven automated theorem provers, DeepHOL, with strong initial results on this benchmark.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
A new language representation model, BERT, designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers, which can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks.
Unified Language Model Pre-training for Natural Language Understanding and Generation
A new Unified pre-trained Language Model (UniLM) that can be fine-tuned for both natural language understanding and generation tasks that compares favorably with BERT on the GLUE benchmark, and the SQuAD 2.0 and CoQA question answering tasks.
Novel positional encodings to enable tree-based transformers
This work abstracts the transformer's sinusoidal positional encodings, allowing it to instead use a novel positional encoding scheme to represent node positions within trees, achieving superior performance over both sequence-to-sequence transformers and state-of-the-art tree-based LSTMs on several datasets.