Instance-aware Prompt Learning for Language Understanding and Generation
@article{Jin2022InstanceawarePL, title={Instance-aware Prompt Learning for Language Understanding and Generation}, author={Feihu Jin and Jinliang Lu and Jiajun Zhang and Chengqing Zong}, journal={ArXiv}, year={2022}, volume={abs/2201.07126} }
Recently, prompt learning has become a new paradigm to utilize pre-trained language models (PLMs) and achieves promising results in downstream tasks with a negligible increase of parameters. The current usage of discrete and continuous prompts assumes that the prompt is fixed for a specific task and all samples in the task share the same prompt. However, a task may contain quite diverse samples in which some are easy and others are difficult, and diverse prompts are desirable. In this paper, we…
Figures and Tables from this paper
10 Citations
PromptGen: Automatically Generate Prompts using Generative Models
- Computer ScienceNAACL-HLT
- 2022
This paper proposes a novel model PromptGen, which can automatically generate prompts conditional on the input sentence, and is the first work considering dynamic prompt generation for knowledge probing, based on a pre-trained generative model.
Vector-Quantized Input-Contextualized Soft Prompts for Natural Language Understanding
- Computer ScienceArXiv
- 2022
VIP particularly focuses on two aspects—contextual prompts that learns input-specific contextualization of the soft prompt tokens through a small-scale sentence encoder and quantized prompts that maps the contextualized prompts to a set of learnable codebook vectors through a Vector quantization network.
Incorporating Instructional Prompts into a Unified Generative Framework for Joint Multiple Intent Detection and Slot Filling
- Computer ScienceCOLING
- 2022
A Unified Generative framework (UGEN) based on a prompt-based paradigm, and formulate the task as a question-answering problem to understand intents, slots, and their implicit correlations is proposed.
Late Prompt Tuning: A Late Prompt Could Be Better Than Many Prompts
- Computer Science
- 2022
Late Prompt Tuning (LPT) is presented that can achieve competitive performance to full model tuning and other PETuning methods under both full-data and few-shot scenarios while possessing faster training speed and lower memory cost.
LingYi: Medical Conversational Question Answering System based on Multi-modal Knowledge Graphs
- Computer ScienceArXiv
- 2022
This paper presents a medical conversational question answering (CQA) system based on the multi-modal knowledge graph, namely “LingYi”, which is designed as a pipeline framework to maintain high visibility and friendly to provide medical services to patients.
UnifieR: A Unified Retriever for Large-Scale Retrieval
- Computer ScienceArXiv
- 2022
A new learning framework, Uni R, is proposed, which combines dense-vector and lexicon-based retrieval in one model with a dual-representing capability, and experiments on passage retrieval benchmarks verify its effectiveness in both paradigms.
Standing on the Shoulders of Giant Frozen Language Models
- Computer ScienceArXiv
- 2022
condense relevant information from 100+ retrieved documents into the input sequence length of the frozen LM reader. We show that can reach and surpass leading fine tuning approaches on Natural…
Prompting through Prototype: A Prototype-based Prompt Learning on Pretrained Vision-Language Models
- Computer ScienceArXiv
- 2022
This work focuses on few-shot image recognition tasks on pretrained vision-language models (PVLMs) and develops a method of prompting through prototype ( PTP), where the image prototype represents a centroid of a certain image cluster in the latent space and a prompt prototype is defined as a soft prompt in the continuous space.
Relation Extraction as Open-book Examination: Retrieval-enhanced Prompt Tuning
- Computer ScienceSIGIR
- 2022
This work regards RE as an open-book examination and proposes a new semiparametric paradigm of retrieval-enhanced prompt tuning for relation extraction, which can achieve state-of-the-art in both standard supervised and few-shot settings.
References
SHOWING 1-10 OF 29 REFERENCES
IDPG: An Instance-Dependent Prompt Generation Method
- Computer ScienceNAACL
- 2022
Extensive experiments on ten natural language understanding (NLU) tasks show that the proposed strategy consistently outperforms various prompt tuning baselines and is on par with other efficient transfer learning methods such as Compacter while tuning far fewer model parameters.
Making Pre-trained Language Models Better Few-shot Learners
- Computer ScienceACL
- 2021
The LM-BFF approach makes minimal assumptions on task resources and domain expertise, and hence constitutes a strong task-agnostic method for few-shot learning.
Prefix-Tuning: Optimizing Continuous Prompts for Generation
- Computer ScienceACL
- 2021
Prefix-tuning is proposed, a lightweight alternative to fine- Tuning for natural language generation tasks, which keeps language model parameters frozen and instead optimizes a sequence of continuous task-specific vectors, which is called the prefix.
The Power of Scale for Parameter-Efficient Prompt Tuning
- Computer ScienceEMNLP
- 2021
This work explores “prompt tuning”, a simple yet effective mechanism for learning “soft prompts” to condition frozen language models to perform specific downstream tasks, and shows that conditioning a frozen model with soft prompts confers benefits in robustness to domain transfer, as compared to full model tuning.
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
- Computer ScienceEACL
- 2021
This work introduces Pattern-Exploiting Training (PET), a semi-supervised training procedure that reformulates input examples as cloze-style phrases to help language models understand a given task.
RobBERT: a Dutch RoBERTa-based Language Model
- Computer ScienceFINDINGS
- 2020
It is found that RobBERT improves state-of-the-art results for various tasks, and especially significantly outperforms other models when dealing with smaller datasets, indicating that it is a powerful pre-trained model for a large variety of Dutch language tasks.
Factual Probing Is [MASK]: Learning vs. Learning to Recall
- Computer ScienceNAACL
- 2021
OptiPrompt is proposed, a novel and efficient method which directly optimizes in continuous embedding space and is able to predict an additional 6.4% of facts in the LAMA benchmark.
Improving and Simplifying Pattern Exploiting Training
- Computer ScienceEMNLP
- 2021
This paper focuses on few-shot learning without any unlabeled data and introduces ADAPET, which modifies PET’s objective to provide denser supervision during fine-tuning and outperforms PET on SuperGLUE without any task-specific unlabeling data.
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
- Computer ScienceNeurIPS
- 2019
A new benchmark styled after GLUE is presented, a new set of more difficult language understanding tasks, a software toolkit, and a public leaderboard are presented.
Transformers: State-of-the-Art Natural Language Processing
- Computer ScienceEMNLP
- 2020
Transformers is an open-source library that consists of carefully engineered state-of-the art Transformer architectures under a unified API and a curated collection of pretrained models made by and available for the community.