A Conversational Paradigm for Program Synthesis

  title={A Conversational Paradigm for Program Synthesis},
  author={Erik Nijkamp and Bo Pang and Hiroaki Hayashi and Lifu Tu and Haiquan Wang and Yingbo Zhou and Silvio Savarese and Caiming Xiong},
Program synthesis strives to generate a computer program as a solution to a given problem specification. We propose a conversational program synthesis approach via large language models, which addresses the challenges of searching over a vast program space and user intent specification faced in prior approaches. Our new approach casts the process of writing a specification and program as a multi-turn conversation between a user and a system. It treats program synthesis as a sequence prediction… 
InCoder: A Generative Model for Code Infilling and Synthesis
INCODER is introduced, a unified generative model that can perform program synthesis (via left-to-right generation) as well as editing (via infilling) and the ability to condition on bidirectional context substantially improves performance on challenging tasks such as type inference, comment generation, and variable re-naming.
ProGen2: Exploring the Boundaries of Protein Language Models
Attention-based models trained on protein sequences have demonstrated incredible success at classification and generation tasks relevant for artificial intelligencedriven protein design. However, we
CERT: Continual Pre-Training on Sketches for Library-Oriented Code Generation
This paper investigates how to leverage an unlabelled code corpus to train a model for library-oriented code generation and presents CERT with two steps: a sketcher generates the sketch, then a generator fills the details in the sketch.
Fault-Aware Neural Code Rankers
Fault-aware neural code rankers are proposed that can predict the correctness of a sampled program without executing it and can significantly increase the pass@1 accuracy of various code generation models on APPS, HumanEval and MBPP datasets.
Formal Specifications from Natural Language
Specifications from example traces and natural language by combining SMT solving and semantic parsing are made by combiningSMT solvingand semantic parsing.
Learning from Self-Sampled Correct and Partially-Correct Programs
This work proposes to let the model perform sampling during training and learn from both self-sampled fully-Correct programs, which yield the gold execution results, as well as partially-correct programs, whose intermediate execution state matches another correct program.
OPT: Open Pre-trained Transformer Language Models
This work presents Open Pre-trained Transformers (OPT), a suite of decoder-only pre-trained transformers ranging from 125M to 175B parameters, which they aim to fully and responsibly share with interested researchers.
In-Context Learning for Few-Shot Dialogue State Tracking
This work proposes an in-context (IC) learning framework for zero-shot and few-shot learning dialogue state tracking (DST), where a large pretrained language model (LM) takes a test instance and a few exemplars as input, and directly decodes the dialogue state without any parameter updates.


Evaluating Large Language Models Trained on Code
It is found that repeated sampling from the GPT language model is a surprisingly effective strategy for producing working solutions to difficult prompts, and the potential broader impacts of deploying powerful code generation technologies, covering safety, security, and economics are discussed.
The Pile: An 800GB Dataset of Diverse Text for Language Modeling
This work presents the Pile, an 825 GiB English text corpus tar-geted at training large-scale language models, constructed from 22 diverse high-quality subsets—both existing and newly constructed—many of which derive from academic or professional sources.
Language Models are Few-Shot Learners
GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic.
JAX: composable transformations of Python+NumPy programs
  • 2018
Program Synthesis with Large Language Models
The limits of the current generation of large language models for program synthesis in general purpose programming languages are explored, finding that even the best models are generally unable to predict the output of a program given a specific input.
The Curious Case of Neural Text Degeneration
By sampling text from the dynamic nucleus of the probability distribution, which allows for diversity while effectively truncating the less reliable tail of the distribution, the resulting text better demonstrates the quality of human text, yielding enhanced diversity without sacrificing fluency and coherence.
Adam: A Method for Stochastic Optimization
This work introduces Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments, and provides a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework.
On the difficulty of training recurrent neural networks
This paper proposes a gradient norm clipping strategy to deal with exploding gradients and a soft constraint for the vanishing gradients problem and validates empirically the hypothesis and proposed solutions.
CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation
Comprehensive experiments show that CodeT5 significantly outperforms prior methods on understanding tasks such as code defect detection and clone detection, and generation tasks across various directions including PL-NL, NL-PL, and PL-PL.