Supporting Clustering with Contrastive Learning

@inproceedings{Zhang2021SupportingCW,
  title={Supporting Clustering with Contrastive Learning},
  author={Dejiao Zhang and Feng Nan and Xiaokai Wei and Shang-Wen Li and Henghui Zhu and Kathleen McKeown and Ramesh Nallapati and Andrew O. Arnold and Bing Xiang},
  booktitle={NAACL},
  year={2021}
}
Unsupervised clustering aims at discovering the semantic categories of data according to some distance measured in the representation space. However, different categories often overlap with each other in the representation space at the beginning of the learning process, which poses a significant challenge for distance-based clustering in achieving good separation between different categories. To this end, we propose Supporting Clustering with Contrastive Learning (SCCL) – a novel framework to… 

Figures and Tables from this paper

You Never Cluster Alone

This paper proposes twin-contrast clustering (TCC), a mainstream contrastive learning paradigm extended to a cluster-level scheme, where all the data subjected to the same cluster contribute to a unified representation that encodes the context of each data group.

Cluster Analysis with Deep Embeddings and Contrastive Learning

This work proposes a novel framework for performing image clustering from deep embeddings by combining instance-level contrastive learning with a deep embedding based cluster center predictor that performs on par with widely accepted clustering methods and outperforms the state-of-the-art Contrastive learning method on the CIFAR-10 dataset.

Pairwise Supervised Contrastive Learning of Sentence Representations

PairSupCon, an instance discrimination based approach aiming to bridge semantic entailment and contradiction understanding with high-level categorical concept encoding, is proposed and evaluated on various downstream tasks that involve understanding sentence semantics at different granularities.

CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization

It is found that the contrastive learning framework consistently produces more factual summaries than strong comparisons with post error correction, entailment-based reranking, and unlikelihood training, according to QA-based factuality evaluation.

MCML: A Novel Memory-based Contrastive Meta-Learning Method for Few Shot Slot Tagging

The Memory-based Contrastive Meta-learning method, a learn-from-memory mechanism that use explicit memory to keep track of the label representations of previously trained episodes and propose a contrastive learning method to compare the current label embedded in the few shot episode with the historic ones stored in the memory.

Theme Transformer: Symbolic Music Generation with Theme-Conditioned Transformer

An alternative conditioning approach is proposed that explicitly trains the Transformer to treat the conditioning sequence as a thematic material that has to manifest itself multiple times in its generation result, and can generate polyphonic pop piano music with repetition and plausible variations of a given condition.

BERT for Russian news clustering

This paper provides results of participation in the Russian News Clustering task within Dialogue Evaluation 2021 and proposes two methods based on BERT for news clustering, one of them shows competitive results in Dialogue 2021 evaluation.

A Comprehensive Survey on Deep Clustering: Taxonomy, Challenges, and Future Directions

A comprehensive survey on deep clustering is conducted by proposing a new taxonomy of different state-of-theart approaches and summarize the essential components of deep clustered and categorize existing methods by the ways they design interactions between deep representation learning and clustering.

New Intent Discovery with Pre-training and Contrastive Learning

This paper proposes a multi-task pre-training strategy to leverage rich unlabeled data along with external labeled data for representation learning, and designs a new contrastive loss to exploit self-supervisory signals in unlabeling data for clustering.

SimCTC: A Simple Contrast Learning Method of Text Clustering (Student Abstract)

Experimental results on multiple benchmark datasets demonstrate that SimCTC remarkably outperforms 6 competitive text clustering methods with 1%-6% improvement on Accuracy (ACC) and 1%-4% improved on Normalized Mutual Information (NMI).

References

SHOWING 1-10 OF 56 REFERENCES

Prototypical Contrastive Learning of Unsupervised Representations

This paper introduces prototypes as latent variables to help find the maximum-likelihood estimation of the network parameters in an Expectation-Maximization framework and proposes ProtoNCE loss, a generalized version of the InfoN CE loss for contrastive learning, which encourages representations to be closer to their assigned prototypes.

Unsupervised Deep Embedding for Clustering Analysis

Deep Embedded Clustering is proposed, a method that simultaneously learns feature representations and cluster assignments using deep neural networks and learns a mapping from the data space to a lower-dimensional feature space in which it iteratively optimizes a clustering objective.

Enhancement of Short Text Clustering by Iterative Classification

The proposed clustering enhancement method not only improves the clustering quality of different baseline clustering methods but also outperforms the state-of-the-art short text clustering method on several short text datasets by a statistically significant margin.

i-Mix: A Strategy for Regularizing Contrastive Representation Learning

It is demonstrated that i-Mix consistently improves the quality of self-supervised representations across domains, resulting in significant performance gains on downstream tasks, and its regularization effect is confirmed via extensive ablation studies across model and dataset sizes.

Supervised Contrastive Learning

A novel training methodology that consistently outperforms cross entropy on supervised learning tasks across different architectures and data augmentations is proposed, and the batch contrastive loss is modified, which has recently been shown to be very effective at learning powerful representations in the self-supervised setting.

A model-based approach for text clustering with outlier detection

This paper proposes a collapsed Gibbs Sampling algorithm for the Dirichlet Process Multinomial Mixture model for text clustering (abbr. to GSDPMM) which does not need to specify the number of clusters in advance and can cope with the high-dimensional problem oftext clustering.

Unsupervised Visual Representation Learning by Context Prediction

It is demonstrated that the feature representation learned using this within-image context indeed captures visual similarity across images and allows us to perform unsupervised visual discovery of objects like cats, people, and even birds from the Pascal VOC 2011 detection dataset.

SpectralNet: Spectral Clustering using Deep Neural Networks

A deep learning approach to spectral clustering that overcomes the major limitations of scalability and generalization of the spectral embedding and applies VC dimension theory to derive a lower bound on the size of SpectralNet.

A Self-Training Approach for Short Text Clustering

The method is proposed, which learns discriminative features from both an autoencoder and a sentence embedding, then uses assignments from a clustering algorithm as supervision to update weights of the encoder network.

Variational Deep Embedding: An Unsupervised and Generative Approach to Clustering

Variational Deep Embedding (VaDE) is proposed, a novel unsupervised generative clustering approach within the framework of Variational Auto-Encoder (VAE), which shows its capability of generating highly realistic samples for any specified cluster, without using supervised information during training.
...