• Corpus ID: 64908139

A Simple but Tough-to-Beat Baseline for Sentence Embeddings

  title={A Simple but Tough-to-Beat Baseline for Sentence Embeddings},
  author={Sanjeev Arora and Yingyu Liang and Tengyu Ma},
Unsupervised Random Walk Sentence Embeddings: A Strong but Simple Baseline
This paper first shows that word vector length has a confounding effect on the probability of a sentence being generated in Arora et al.
Sentence Embeddings using Supervised Contrastive Learning
This paper fine-tunes pretrained BERT on SNLI data, incorporating both supervised crossentropy loss and supervised contrastive loss, and proposes a new method to build sentence embeddings by doing supervised Contrastive learning.
Sequential Sentence Embeddings for Semantic Similarity
This work proposes a novel approach to compute sentence embeddings for semantic similarity that exploits a linear autoencoder for sequences and provides a grounded approach to identify and subtract common discourse from a sentence and its embedding, to remove associated uninformative features.
MappSent: a Textual Mapping Approach for Question-to-Question Similarity
MappSent is introduced, a novel approach for textual similarity based on a linear sentence embedding representation that maps sentences in a joint-subspace where similar sets of sentences are pushed closer and achieves competitive results and outperforms in most cases state-of-art methods.
A La Carte Embedding: Cheap but Effective Induction of Semantic Feature Vectors
A la carte embedding is introduced, a simple and general alternative to the usual word2vec-based approaches for building such representations that is based upon recent theoretical results for GloVe-like embeddings.
Meta-Embeddings for Natural Language Inference and Semantic Similarity tasks
Meta Embedding derived from few State-of-the-Art (SOTA) models are proposed to efficiently tackle mainstream NLP tasks like classification, semantic relatedness, and text similarity by showing us that meta-embeddings can be used for several NLP task by harnessing the power of several individual representations.
Evaluation of sentence embeddings in downstream and linguistic probing tasks
It is shown that a simple approach using bag-of-words with a recently introduced language model for deep context-dependent word embeddings proved to yield better results in many tasks when compared to sentence encoders trained on entailment datasets.
On the Sentence Embeddings from Pre-trained Language Models
This paper proposes to transform the anisotropic sentence embedding distribution to a smooth and isotropic Gaussian distribution through normalizing flows that are learned with an unsupervised objective and achieves significant performance gains over the state-of-the-art sentence embeddings on a variety of semantic textual similarity tasks.
Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms
This paper conducts a point-by-point comparative study between Simple Word-Embedding-based Models (SWEMs), consisting of parameter-free pooling operations, relative to word-embedding-based RNN/CNN models, and proposes two additional pooling strategies over learned word embeddings: a max-pooling operation for improved interpretability and a hierarchical pooling operation, which preserves spatial information within text sequences.
An Empirical Study on Post-processing Methods for Word Embeddings
By optimising an objective in the semi-Riemannian manifold with Centralised Kernel Alignment, this work is able to search for the optimal shrinkage parameter, and provide a post-processing method to smooth the spectrum of learnt word vectors which yields improved performance on downstream tasks.