Learning Contextual Representations for Semantic Parsing with Generation-Augmented Pre-Training

@inproceedings{Shi2021LearningCR,
  title={Learning Contextual Representations for Semantic Parsing with Generation-Augmented Pre-Training},
  author={Peng Shi and Patrick Ng and Zhiguo Wang and Henghui Zhu and Alexander Hanbo Li and Jun Wang and C{\'i}cero Nogueira dos Santos and Bing Xiang},
  booktitle={AAAI},
  year={2021}
}
Most recently, there has been significant interest in learning contextual representations for various NLP tasks, by leveraging large scale text corpora to train powerful language models with self-supervised learning objectives, such as Masked Language Model (MLM). Based on a pilot study, we observe three issues of existing general-purpose language models when they are applied in the text-to-SQL semantic parsers: fail to detect the column mentions in the utterances, to infer the column mentions… 

Figures and Tables from this paper

Linking-Enhanced Pre-Training for Table Semantic Parsing

TLDR
Two novel pre-training objectives are designed to impose the desired inductive bias into the learned representations for table pre- Training and a schema-aware curriculum learning approach is proposed to mitigate the impact of noise and learn effectively from the pre- training data in an easy-to-hard manner.

SDCUP: Schema Dependency-Enhanced Curriculum Pre-Training for Table Semantic Parsing

TLDR
Two novel pre-training objectives are designed to impose the desired inductive bias into the learned representations for table pre- Training and a schema-aware curriculum learning approach is proposed to mitigate the impact of noise and learn effectively from the pre- training data in an easy-to-hard manner.

Generation-Focused Table-Based Intermediate Pre-training for Free-Form Question Answering

TLDR
An intermediate pre-training framework, Generation-focused Table-based Intermediate Pre-training (GENTAP), that jointly learns representations of natural language questions and tables that enhance the question understanding and table representation abilities for complex questions is presented.

Relation Aware Semi-autoregressive Semantic Parsing for NL2SQL

TLDR
A Relation aware Semi-autogressive Semantic Parsing (RaSaP) framework is presented, which is more adaptable for NL2 SQL and shows its effectiveness in learning better word representation in NL2SQL.

Data Augmentation with Hierarchical SQL-to-Question Generation for Cross-domain Text-to-SQL Parsing

TLDR
Experiments on three cross-domain datasets show that the proposed data augmentation framework can consistently improve performance over strong baselines, and the hierarchical generation component is the key for the improvement.

Measuring and Improving Compositional Generalization in Text-to-SQL via Component Alignment

TLDR
A number of state-of-the-art models are modified to train on the segmented data of Spider-SS, and it is shown that this method improves the generalization performance.

Hierarchical Neural Data Synthesis for Semantic Parsing

TLDR
This work proposes a purely neural approach of data augmentation for semantic parsing that completely removes the need for grammar engineering while achieving higher semantic parsing accuracy on the Spider cross-domain text-toSQL semantic parsing benchmark.

Decoupled Dialogue Modeling and Semantic Parsing for Multi-Turn Text-to-SQL

TLDR
This paper proposes a novel decoupled multi-turn Text-to-SQL framework, where an utterance rewrite model first explicitly solves completion of dialogue context, and then a single-turn textscript parser follows to address the data sparsity problem.

Testing Cross-Database Semantic Parsers Using Canonical Utterances

TLDR
This work characterized a set of essential capabilities for cross-database semantic parsing models, and detailed the method for synthesizing the corresponding test data, and evaluated a variety of high performing models using the proposed approach.

A Survey on Text-to-SQL Parsing: Concepts, Methods, and Future Directions

TLDR
A comprehensive review on deep learning approaches for text-to-SQL parsing with a systematical overview of pre-trained language models and existing methods and some potential future directions are explored.

References

SHOWING 1-10 OF 48 REFERENCES

GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing

TLDR
GraPPa is an effective pre-training approach for table semantic parsing that learns a compositional inductive bias in the joint representations of textual and tabular data and significantly outperforms RoBERTa-large as the feature representation layers and establishes new state-of-the-art results on all of them.

TaPas: Weakly Supervised Table Parsing via Pre-training

TLDR
TaPas is presented, an approach to question answering over tables without generating logical forms that outperforms or rivals semantic parsing models by improving state-of-the-art accuracy on SQA and performing on par with the state of theart on WikiSQL and WikiTQ, but with a simpler model architecture.

TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data

TLDR
TaBERT is a pretrained LM that jointly learns representations for NL sentences and (semi-)structured tables that achieves new best results on the challenging weakly-supervised semantic parsing benchmark WikiTableQuestions, while performing competitively on the text-to-SQL dataset Spider.

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

TLDR
This systematic study compares pre-training objectives, architectures, unlabeled datasets, transfer approaches, and other factors on dozens of language understanding tasks and achieves state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more.

Linguistic Knowledge and Transferability of Contextual Representations

TLDR
It is found that linear models trained on top of frozen contextual representations are competitive with state-of-the-art task-specific models in many cases, but fail on tasks requiring fine-grained linguistic knowledge.

Knowledge Enhanced Contextual Word Representations

TLDR
After integrating WordNet and a subset of Wikipedia into BERT, the knowledge enhanced BERT (KnowBert) demonstrates improved perplexity, ability to recall facts as measured in a probing task and downstream performance on relationship extraction, entity typing, and word sense disambiguation.

DART: Open-Domain Structured Data Record to Text Generation

TLDR
The dataset construction framework effectively merged heterogeneous sources from open domain semantic parsing and spoken dialogue systems by utilizing techniques including tree ontology annotation, question-answer pair to declarative sentence conversion, and predicate unification, all with minimum post-editing.

Unified Language Model Pre-training for Natural Language Understanding and Generation

TLDR
A new Unified pre-trained Language Model (UniLM) that can be fine-tuned for both natural language understanding and generation tasks that compares favorably with BERT on the GLUE benchmark, and the SQuAD 2.0 and CoQA question answering tasks.

Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model

TLDR
This work proposes a simple yet effective weakly supervised pretraining objective, which explicitly forces the model to incorporate knowledge about real-world entities, and consistently outperforms BERT on four entity-related question answering datasets.

Reranking for Neural Semantic Parsing

TLDR
This paper presents a simple approach to quickly iterate and improve the performance of an existing neural semantic parser by reranking an n-best list of predicted MRs, using features that are designed to fix observed problems with baseline models.