• Corpus ID: 246036341

Instance-aware Prompt Learning for Language Understanding and Generation

@article{Jin2022InstanceawarePL,
  title={Instance-aware Prompt Learning for Language Understanding and Generation},
  author={Feihu Jin and Jinliang Lu and Jiajun Zhang and Chengqing Zong},
  journal={ArXiv},
  year={2022},
  volume={abs/2201.07126}
}
Recently, prompt learning has become a new paradigm to utilize pre-trained language models (PLMs) and achieves promising results in downstream tasks with a negligible increase of parameters. The current usage of discrete and continuous prompts assumes that the prompt is fixed for a specific task and all samples in the task share the same prompt. However, a task may contain quite diverse samples in which some are easy and others are difficult, and diverse prompts are desirable. In this paper, we… 
[PDF] Semantic Reader
10 Citations
Highly Influential Citations
1
Background Citations
8
Methods Citations
3

Figures and Tables from this paper

10 Citations

PromptGen: Automatically Generate Prompts using Generative Models

This paper proposes a novel model PromptGen, which can automatically generate prompts conditional on the input sentence, and is the first work considering dynamic prompt generation for knowledge probing, based on a pre-trained generative model.

Vector-Quantized Input-Contextualized Soft Prompts for Natural Language Understanding

VIP particularly focuses on two aspects—contextual prompts that learns input-specific contextualization of the soft prompt tokens through a small-scale sentence encoder and quantized prompts that maps the contextualized prompts to a set of learnable codebook vectors through a Vector quantization network.

Incorporating Instructional Prompts into a Unified Generative Framework for Joint Multiple Intent Detection and Slot Filling

A Unified Generative framework (UGEN) based on a prompt-based paradigm, and formulate the task as a question-answering problem to understand intents, slots, and their implicit correlations is proposed.

Late Prompt Tuning: A Late Prompt Could Be Better Than Many Prompts

Late Prompt Tuning (LPT) is presented that can achieve competitive performance to full model tuning and other PETuning methods under both full-data and few-shot scenarios while possessing faster training speed and lower memory cost.

LingYi: Medical Conversational Question Answering System based on Multi-modal Knowledge Graphs

This paper presents a medical conversational question answering (CQA) system based on the multi-modal knowledge graph, namely “LingYi”, which is designed as a pipeline framework to maintain high visibility and friendly to provide medical services to patients.

UnifieR: A Unified Retriever for Large-Scale Retrieval

A new learning framework, Uni R, is proposed, which combines dense-vector and lexicon-based retrieval in one model with a dual-representing capability, and experiments on passage retrieval benchmarks verify its effectiveness in both paradigms.

Standing on the Shoulders of Giant Frozen Language Models

condense relevant information from 100+ retrieved documents into the input sequence length of the frozen LM reader. We show that can reach and surpass leading fine tuning approaches on Natural…

Prompting through Prototype: A Prototype-based Prompt Learning on Pretrained Vision-Language Models

This work focuses on few-shot image recognition tasks on pretrained vision-language models (PVLMs) and develops a method of prompting through prototype ( PTP), where the image prototype represents a centroid of a certain image cluster in the latent space and a prompt prototype is defined as a soft prompt in the continuous space.

Relation Extraction as Open-book Examination: Retrieval-enhanced Prompt Tuning

This work regards RE as an open-book examination and proposes a new semiparametric paradigm of retrieval-enhanced prompt tuning for relation extraction, which can achieve state-of-the-art in both standard supervised and few-shot settings.

FROZEN LANGUAGE MODELS

  • 2022

References

SHOWING 1-10 OF 29 REFERENCES

IDPG: An Instance-Dependent Prompt Generation Method

Extensive experiments on ten natural language understanding (NLU) tasks show that the proposed strategy consistently outperforms various prompt tuning baselines and is on par with other efficient transfer learning methods such as Compacter while tuning far fewer model parameters.

Making Pre-trained Language Models Better Few-shot Learners

The LM-BFF approach makes minimal assumptions on task resources and domain expertise, and hence constitutes a strong task-agnostic method for few-shot learning.

Prefix-Tuning: Optimizing Continuous Prompts for Generation

Prefix-tuning is proposed, a lightweight alternative to fine- Tuning for natural language generation tasks, which keeps language model parameters frozen and instead optimizes a sequence of continuous task-specific vectors, which is called the prefix.

The Power of Scale for Parameter-Efficient Prompt Tuning

This work explores “prompt tuning”, a simple yet effective mechanism for learning “soft prompts” to condition frozen language models to perform specific downstream tasks, and shows that conditioning a frozen model with soft prompts confers benefits in robustness to domain transfer, as compared to full model tuning.

Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference

This work introduces Pattern-Exploiting Training (PET), a semi-supervised training procedure that reformulates input examples as cloze-style phrases to help language models understand a given task.

RobBERT: a Dutch RoBERTa-based Language Model

It is found that RobBERT improves state-of-the-art results for various tasks, and especially significantly outperforms other models when dealing with smaller datasets, indicating that it is a powerful pre-trained model for a large variety of Dutch language tasks.

Factual Probing Is [MASK]: Learning vs. Learning to Recall

OptiPrompt is proposed, a novel and efficient method which directly optimizes in continuous embedding space and is able to predict an additional 6.4% of facts in the LAMA benchmark.

Improving and Simplifying Pattern Exploiting Training

This paper focuses on few-shot learning without any unlabeled data and introduces ADAPET, which modifies PET’s objective to provide denser supervision during fine-tuning and outperforms PET on SuperGLUE without any task-specific unlabeling data.

SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems

A new benchmark styled after GLUE is presented, a new set of more difficult language understanding tasks, a software toolkit, and a public leaderboard are presented.

Transformers: State-of-the-Art Natural Language Processing

Transformers is an open-source library that consists of carefully engineered state-of-the art Transformer architectures under a unified API and a curated collection of pretrained models made by and available for the community.