• Corpus ID: 235377032

Energy-Based Models for Code Generation under Compilability Constraints

@article{Korbak2021EnergyBasedMF,
  title={Energy-Based Models for Code Generation under Compilability Constraints},
  author={Tomasz Korbak and Hady ElSahar and Marc Dymetman and Germ{\'a}n Kruszewski},
  journal={ArXiv},
  year={2021},
  volume={abs/2106.04985}
}
Neural language models can be successfully trained on source code, leading to applications such as code completion. However, their versatile autoregressive self-supervision objective overlooks important global sequence-level features that are present in the data such as syntactic correctness or compilability. In this work, we pose the problem of learning to generate compilable code as constraint satisfaction. We define an Energy-Based Model (EBM) representing a pre-trained generative model with… 
View PDF on arXiv
View With Semantic Reader
6 Citations
Background Citations
4
Methods Citations
4

6 Citations

Compilable Neural Code Generation with Compiler Feedback
TLDR
To improve compilability of the generated programs, this paper proposes COMPCODER, a three-stage pipeline utilizing compiler feedback for compilable code generation, including language model fine-tuning,Compilability reinforcement, and compilabilities discrimination.
Controlling Conditional Language Models with Distributional Policy Gradients
TLDR
The results show that fine-tuning using CDPG robustly moves these pretrained models closer towards meeting control objectives and — in contrast with baseline approaches — does not result in catastrophic forgetting.
On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting
TLDR
The theoretical connections between the two paradigms, and it is shown that methods such as KL-control developed for RM can also be construed as belonging to DM, are explored, and that while DM differs from RM, it can suffer from similar training difficulties, such as high gradient variance.
RL with KL penalties is better viewed as Bayesian inference
TLDR
This paper analyzes challenges associated with treating a language model as an RL policy and shows how avoiding those challenges requires moving beyond the RL paradigm, and shows that KL-regularised RL is equivalent to variational inference: approximating a Bayesian posterior which informs how to update a prior LM to conform with evidence provided by the reward function.
AXIMIZATION AND D ISTRIBUTION INE -T UNING L
  • Computer Science
  • 2021
TLDR
The intimate connections between the two paradigms are explored, and it is shown that methods such as KL-control developed in the RM paradigm can be construed as belonging to DM, and that while DM differs from RM, it can suffer from similar training difficulties, such as high gradient variance.
Controlling Conditional Language Models without Catastrophic Forgetting
TLDR
DPG is extended to conditional tasks by proposing Conditional DPG (CDPG), and results show thatne-tuning using CDPG robustly moves these pretrained models closer towards meeting control objectives and — in contrast with baseline approaches — does not result in catastrophic forgetting.

References

SHOWING 1-10 OF 42 REFERENCES
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
TLDR
This paper introduces CodeXGLUE, a benchmark dataset to foster machine learning research for program understanding and generation that includes a collection of 10 tasks across 14 datasets and a platform for model evaluation and comparison.
Structured Generative Models of Natural Source Code
TLDR
A family of generative models for NSC that have three key properties: first, they incorporate both sequential and hierarchical structure, second, they learn a distributed representation of source code elements, and third, they integrate closely with a compiler.
An Empirical Study on the Usage of BERT Models for Code Completion
TLDR
A large-scale empirical study aimed at exploring the capabilities of state-of-the-art deep learning (DL) models in supporting code completion at different granularity levels, including single tokens, one or multiple entire statements, up to entire code blocks.
PHOG: Probabilistic Model for Code
TLDR
PHOG generalizes probabilistic context free grammars (PCFGs) by allowing conditioning of a production rule beyond the parent non-terminal, thus capturing rich contexts relevant to programs.
HuggingFace's Transformers: State-of-the-art Natural Language Processing
TLDR
The \textit{Transformers} library is an open-source library that consists of carefully engineered state-of-the art Transformer architectures under a unified API and a curated collection of pretrained models made by and available for the community.
A Survey of Machine Learning for Big Code and Naturalness
TLDR
This article presents a taxonomy based on the underlying design principles of each model and uses it to navigate the literature and discuss cross-cutting and application-specific challenges and opportunities.
SPoC: Search-based Pseudocode to Code
TLDR
This work proposes to perform credit assignment based on signals from compilation errors, which constitute 88.7% of program failures and improves the synthesis success rate over using the top-one translation of the pseudocode from 25.6% to 44.7%.
Language Models are Unsupervised Multitask Learners
TLDR
It is demonstrated that language models begin to learn these tasks without any explicit supervision when trained on a new dataset of millions of webpages called WebText, suggesting a promising path towards building language processing systems which learn to perform tasks from their naturally occurring demonstrations.
A Deep Reinforced Model for Abstractive Summarization
TLDR
A neural network model with a novel intra-attention that attends over the input and continuously generated output separately, and a new training method that combines standard supervised word prediction and reinforcement learning (RL) that produces higher quality summaries.
Visualizing and Understanding Recurrent Networks
TLDR
This work uses character-level language models as an interpretable testbed to provide an analysis of LSTM representations, predictions and error types, and reveals the existence of interpretable cells that keep track of long-range dependencies such as line lengths, quotes and brackets.
...
...