In-IDE Code Generation from Natural Language: Promise and Challenges

@article{Xu2022InIDECG,
  title={In-IDE Code Generation from Natural Language: Promise and Challenges},
  author={Frank F. Xu and Bogdan Vasilescu and Graham Neubig},
  journal={ACM Transactions on Software Engineering and Methodology (TOSEM)},
  year={2022},
  volume={31},
  pages={1 - 47}
}
A great part of software development involves conceptualizing or communicating the underlying procedures and logic that needs to be expressed in programs. One major difficulty of programming is turning concept into code, especially when dealing with the APIs of unfamiliar libraries. Recently, there has been a proliferation of machine learning methods for code generation and retrieval from natural language queries, but these have primarily been evaluated purely based on retrieval accuracy or… 
Natural language-guided programming
TLDR
The key idea is to adapt code autocompletion tools such that they take into account not only the developer’s already-written code but also the intent of the task the developer is trying to achieve next, formulated in plain natural language.
Better Together? An Evaluation of AI-Supported Code Translation
TLDR
This work motivates the need for intelligent user interfaces that help software engineers effectively work with generative code models in order to understand and evaluate their outputs and achieve superior outcomes to working alone.
Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models
TLDR
It was found that, while Copilot did not necessarily improve the task completion time or success rate, most participants preferred to use Copilot in daily programming tasks, since Copilot often provided a useful starting point and saved the effort of searching online.
Evaluating Large Language Models Trained on Code
TLDR
It is found that repeated sampling from the GPT language model is a surprisingly effective strategy for producing working solutions to difficult prompts, and the potential broader impacts of deploying powerful code generation technologies, covering safety, security, and economics are discussed.
Discovering the Syntax and Strategies of Natural Language Programming with Generative Language Models
TLDR
A natural language code synthesis tool, GenLine, backed by a large generative language model and a set of task-specific prompts that create or change code is presented, indicating that while naturallanguage code synthesis can sometimes provide a magical experience, participants still faced challenges.
Toward Code Generation: A Survey and Lessons from Semantic Parsing
TLDR
An overview of the growing body of research in semantic parsing works from an evolutionary perspective, with specific analyses on neuro-symbolic methods, architecture, and supervision is presented.
Retrieval Augmented Code Generation and Summarization
TLDR
A retrieval augmented framework that retrieves relevant code or summaries from a retrieval database and provides them as a supplement to code generation or summarization models, and the promising results endorse the effectiveness of the proposed REDCODER.
S COTCH : A S EMANTIC C ODE S EARCH E NGINE FOR IDE S
TLDR
Results from automated as well as human evaluation suggest that the inclusion of code context in search significantly improves the retrieval of the correct code snippet but slightly impairs ranking quality among code snippets.
NLI-GSC: A Natural Language Interface for Generating SourceCode
TLDR
Another novel method has been proposed to create dataset from scratch using predefined structure that is filled with predefined keywords creating unique combination of training dataset.
Synchromesh: Reliable code generation from pre-trained language models
TLDR
This paper proposes SYNCHROMESH: a framework for substantially improving the reliability of pre-trained models for code generation, and observes substantial complementary gains from CSD and TST in prediction accuracy and in effectively preventing run-time errors.
...
...

References

SHOWING 1-10 OF 147 REFERENCES
Mapping Language to Code in Programmatic Context
TLDR
This work introduces CONCODE, a new large dataset with over 100,000 examples consisting of Java classes from online code repositories, and develops a new encoder-decoder architecture that models the interaction between the method documentation and the class environment.
ANNE: Improving Source Code Search using Entity Retrieval Approach
TLDR
This work presents a programming language agnostic technique to discover mappings between syntactic forms and natural language terms representing programming concepts, and uses the questions and answers in Stack Overflow to create this mapping.
Incorporating External Knowledge through Pre-training for Natural Language to Code Generation
TLDR
Evaluations show that combining the two sources with data augmentation and retrieval-based data re-sampling improves the current state-of-the-art by up to 2.2% absolute BLEU score on the code generation testbed CoNaLa.
Query expansion via WordNet for effective code search
TLDR
This work proposes an approach that extends a query with synonyms generated from WordNet that improves the precision and recall of Conquer, a state-of-the-art query expansion/reformulation technique, by 5% and 8% respectively.
CodeSearchNet Challenge: Evaluating the State of Semantic Code Search
TLDR
The methodology used to obtain the corpus and expert labels, as well as a number of simple baseline solutions for the task are described.
Mining StackOverflow to turn the IDE into a self-confident programming prompter
TLDR
A novel approach is proposed that, given a context in the IDE, automatically retrieves pertinent discussions from Stack Overflow, evaluates their relevance, and, if a given confidence threshold is surpassed, notifies the developer about the available help.
Retrieval-Based Neural Code Generation
TLDR
Recode, a method based on subtree retrieval that makes it possible to explicitly reference existing code examples within a neural code generation model, is introduced.
Code completion with statistical language models
TLDR
The main idea is to reduce the problem of code completion to a natural-language processing problem of predicting probabilities of sentences, and design a simple and scalable static analysis that extracts sequences of method calls from a large codebase, and index these into a statistical language model.
SWIM: Synthesizing What I Mean - Code Search and Idiomatic Snippet Synthesis
TLDR
SWIM, a tool which suggests code snippets given API-related natural language queries such as "generate md5 hash code" is described, which translates user queries into the APIs of interest using clickthrough data from the Bing search engine.
Program Synthesis and Semantic Parsing with Learned Code Idioms
TLDR
PATOIS is a system that allows a neural program synthesizer to explicitly interleave high-level and low-level reasoning at every generation step, and it accomplishes this by automatically mining common code idioms from a given corpus and incorporating them into the underlying language for neural synthesis.
...
...