• Corpus ID: 246036341

Instance-aware Prompt Learning for Language Understanding and Generation

@article{Jin2022InstanceawarePL,
  title={Instance-aware Prompt Learning for Language Understanding and Generation},
  author={Feihu Jin and Jinliang Lu and Jiajun Zhang and Chengqing Zong},
  journal={ArXiv},
  year={2022},
  volume={abs/2201.07126}
}
Recently, prompt learning has become a new paradigm to utilize pre-trained language models (PLMs) and achieves promising results in downstream tasks with a negligible increase of parameters. The current usage of discrete and continuous prompts assumes that the prompt is fixed for a specific task and all samples in the task share the same prompt. However, a task may contain quite diverse samples in which some are easy and others are difficult, and diverse prompts are desirable. In this paper, we… 
[PDF] Semantic Reader
10 Citations
Highly Influential Citations
1
Background Citations
8
Methods Citations
3

Figures and Tables from this paper

10 Citations

Prompting through Prototype: A Prototype-based Prompt Learning on Pretrained Vision-Language Models

This work focuses on few-shot image recognition tasks on pretrained vision-language models (PVLMs) and develops a method of prompting through prototype ( PTP), where the image prototype represents a centroid of a certain image cluster in the latent space and a prompt prototype is defined as a soft prompt in the continuous space.

PromptGen: Automatically Generate Prompts using Generative Models

This paper proposes a novel model PromptGen, which can automatically generate prompts conditional on the input sentence, and is the first work considering dynamic prompt generation for knowledge probing, based on a pre-trained generative model.

Vector-Quantized Input-Contextualized Soft Prompts for Natural Language Understanding

This work proposes a novel way of prompting, Vector-quantized Input-contextualized Prompt Tuning or VIP, which focuses on two aspects i) input-adaptation: input-specific contextualization of the soft tokens; and ii) vector quantization: the authors pass the tokens through a quantizer which effectively reduces representation variance by sampling prompts from a compact latent space.

Incorporating Instructional Prompts into a Unified Generative Framework for Joint Multiple Intent Detection and Slot Filling

A Unified Generative framework (UGEN) based on a prompt-based paradigm, and formulate the task as a question-answering problem to understand intents, slots, and their implicit correlations is proposed.

Relation Extraction as Open-book Examination: Retrieval-enhanced Prompt Tuning

This work regards RE as an open-book examination and proposes a new semiparametric paradigm of retrieval-enhanced prompt tuning for relation extraction, which can achieve state-of-the-art in both standard supervised and few-shot settings.

Late Prompt Tuning: A Late Prompt Could Be Better Than Many Prompts

Late Prompt Tuning (LPT) is presented that can achieve competitive performance to full model tuning and other PETuning methods under both full-data and few-shot scenarios while possessing faster training speed and lower memory cost.

LingYi: Medical Conversational Question Answering System based on Multi-modal Knowledge Graphs

This paper presents a medical conversational question answering (CQA) system based on the multi-modal knowledge graph, namely “LingYi”, which is designed as a pipeline framework to maintain high visibility and friendly to provide medical services to patients.

UnifieR: A Unified Retriever for Large-Scale Retrieval

A new learning framework, Uni R, is proposed, which combines dense-vector and lexicon-based retrieval in one model with a dual-representing capability, and experiments on passage retrieval benchmarks verify its effectiveness in both paradigms.

FROZEN LANGUAGE MODELS

  • 2022

Standing on the Shoulders of Giant Frozen Language Models

condense relevant information from 100+ retrieved documents into the input sequence length of the frozen LM reader. We show that can reach and surpass leading fine tuning approaches on Natural…

References

SHOWING 1-10 OF 29 REFERENCES

Making Pre-trained Language Models Better Few-shot Learners

The LM-BFF approach makes minimal assumptions on task resources and domain expertise, and hence constitutes a strong task-agnostic method for few-shot learning.

Prefix-Tuning: Optimizing Continuous Prompts for Generation

Prefix-tuning is proposed, a lightweight alternative to fine- Tuning for natural language generation tasks, which keeps language model parameters frozen and instead optimizes a sequence of continuous task-specific vectors, which is called the prefix.

The Power of Scale for Parameter-Efficient Prompt Tuning

This work explores “prompt tuning”, a simple yet effective mechanism for learning “soft prompts” to condition frozen language models to perform specific downstream tasks, and shows that conditioning a frozen model with soft prompts confers benefits in robustness to domain transfer, as compared to full model tuning.

Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference

This work introduces Pattern-Exploiting Training (PET), a semi-supervised training procedure that reformulates input examples as cloze-style phrases to help language models understand a given task.

RobBERT: a Dutch RoBERTa-based Language Model

It is found that RobBERT improves state-of-the-art results for various tasks, and especially significantly outperforms other models when dealing with smaller datasets, indicating that it is a powerful pre-trained model for a large variety of Dutch language tasks.

Factual Probing Is [MASK]: Learning vs. Learning to Recall

OptiPrompt is proposed, a novel and efficient method which directly optimizes in continuous embedding space and is able to predict an additional 6.4% of facts in the LAMA benchmark.

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

This work presents two parameter-reduction techniques to lower memory consumption and increase the training speed of BERT, and uses a self-supervised loss that focuses on modeling inter-sentence coherence.

Improving and Simplifying Pattern Exploiting Training

This paper focuses on few-shot learning without any unlabeled data and introduces ADAPET, which modifies PET’s objective to provide denser supervision during fine-tuning and outperforms PET on SuperGLUE without any task-specific unlabeling data.

DART: Open-Domain Structured Data Record to Text Generation

The dataset construction framework effectively merged heterogeneous sources from open domain semantic parsing and spoken dialogue systems by utilizing techniques including tree ontology annotation, question-answer pair to declarative sentence conversion, and predicate unification, all with minimum post-editing.

SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems

A new benchmark styled after GLUE is presented, a new set of more difficult language understanding tasks, a software toolkit, and a public leaderboard are presented.