# Noise Contrastive Meta-Learning for Conditional Density Estimation using Kernel Mean Embeddings

@inproceedings{Ton2021NoiseCM, title={Noise Contrastive Meta-Learning for Conditional Density Estimation using Kernel Mean Embeddings}, author={Jean-Francois Ton and Lucian Chan and Yee Whye Teh and D. Sejdinovic}, booktitle={AISTATS}, year={2021} }

Current meta-learning approaches focus on learning functional representations of relationships between variables, i.e. on estimating conditional expectations in regression. In many applications, however, we are faced with conditional distributions which cannot be meaningfully summarized using expectation only (due to e.g. multimodality). Hence, we consider the problem of conditional density estimation in the meta-learning setting. We introduce a novel technique for meta-learning which combines…

## Figures, Tables, and Topics from this paper

## 7 Citations

Kernel Conditional Density Operators

- Computer Science, MathematicsAISTATS
- 2020

We introduce a novel conditional density estimation model termed the conditional density operator (CDO). It naturally captures multivariate, multimodal output densities and shows performance that is…

Score Matched Neural Exponential Families for Likelihood-Free Inference

- Mathematics
- 2020

Bayesian Likelihood-Free Inference (LFI) approaches allow to obtain posterior distributions for stochastic models with intractable likelihood, by relying on model simulations. In Approximate Bayesian…

BayesIMP: Uncertainty Quantification for Causal Data Fusion

- Computer Science, MathematicsArXiv
- 2021

Bayesian Interventional Mean Processes is introduced, a framework which combines ideas from probabilistic integration and kernel mean embeddings to represent interventional distributions in the reproducing kernel Hilbert space, while taking into account the uncertainty within each causal graph.

Function Contrastive Learning of Transferable Meta-Representations

- Computer Science, MathematicsICML
- 2021

A decoupled encoder-decoder approach to supervised meta-learning, where the encoder is trained with a contrastive objective to find a good representation of the underlying function and the representations obtained outperform strong baselines in terms of downstream performance and noise robustness.

Meta Two-Sample Testing: Learning Kernels for Testing with Limited Data

- Mathematics, Computer ScienceArXiv
- 2021

This work provides both theoretical justification and empirical evidence that the proposed meta-testing schemes outperform learning kernel-based tests directly from scarce observations, and identifies when such schemes will be successful.

On Contrastive Representations of Stochastic Processes

- Computer Science, MathematicsArXiv
- 2021

A unifying framework for learning contrastive representations of stochastic processes (CRESP) that does away with exact reconstruction, and whose methods tolerate noisy high-dimensional observations better than traditional approaches, and the learned representations transfer to a range of downstream tasks.

Score Matched Conditional Exponential Families for Likelihood-Free Inference∗

- 2020

To perform Bayesian inference for stochastic simulator models for which the likelihood is not accessible, Likelihood-Free Inference (LFI) relies on simulations from the model. Standard LFI methods…

## References

SHOWING 1-10 OF 37 REFERENCES

Conditional mean embeddings as regressors - supplementary

- Mathematics, Computer ScienceArXiv
- 2012

This work demonstrates an equivalence between reproducing kernel Hilbert space (RKHS) embeddings of conditional distributions and vector-valued regressors and derives minimax convergence rates which are valid under milder and more intuitive assumptions.

Gaussian Process Conditional Density Estimation

- Computer Science, MathematicsNeurIPS
- 2018

This work proposes to extend the model's input with latent variables and use Gaussian processes to map this augmented input onto samples from the conditional distribution, and illustrates the effectiveness and wide-reaching applicability of the model on a variety of real-world problems.

Kernel Exponential Family Estimation via Doubly Dual Embedding

- Mathematics, Computer ScienceAISTATS
- 2019

A connection between kernel exponential family estimation and MMD-GANs is established, revealing a new perspective for understanding GANs and it is shown that the proposed estimator empirically outperforms state-of-the-art estimators.

Representation Learning with Contrastive Predictive Coding

- Computer Science, MathematicsArXiv
- 2018

This work proposes a universal unsupervised learning approach to extract useful representations from high-dimensional data, which it calls Contrastive Predictive Coding, and demonstrates that the approach is able to learn useful representations achieving strong performance on four distinct domains: speech, images, text and reinforcement learning in 3D environments.

Deep Kernel Learning

- Computer Science, MathematicsAISTATS
- 2016

We introduce scalable deep kernels, which combine the structural properties of deep learning architectures with the non-parametric flexibility of kernel methods. Specifically, we transform the inputs…

Least-Squares Conditional Density Estimation

- Mathematics, Computer ScienceIEICE Trans. Inf. Syst.
- 2010

A novel method of conditional density estimation that is suitable for multi-dimensional continuous variables and expresses the conditional density in terms of the density ratio and the ratio is directly estimated without going through density estimation.

Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiency

- Computer Science, MathematicsEMNLP
- 2018

It is shown that the ranking-based variant of NCE gives consistent parameter estimates under weaker assumptions than the classification-based method, which is closely related to negative sampling methods, now widely used in NLP.

Kernel Mean Embedding of Distributions: A Review and Beyonds

- Mathematics, Computer ScienceFound. Trends Mach. Learn.
- 2017

A comprehensive review of existing work and recent advances in the Hilbert space embedding of distributions, and to discuss the most challenging issues and open problems that could lead to new research directions.

Empirical representations of probability distributions via kernel mean embeddings

- Mathematics
- 2016

How to represent probability distributions is a fundamental issue in statistical methods, as this determines all the subsequent estimation procedures. In this thesis, we focus on the approach using…

A Theoretical Analysis of Contrastive Unsupervised Representation Learning

- Computer Science, MathematicsICML
- 2019

This framework allows us to show provable guarantees on the performance of the learned representations on the average classification task that is comprised of a subset of the same set of latent classes and shows that learned representations can reduce (labeled) sample complexity on downstream tasks.