• Corpus ID: 235422444

Meta Two-Sample Testing: Learning Kernels for Testing with Limited Data

  title={Meta Two-Sample Testing: Learning Kernels for Testing with Limited Data},
  author={Feng Liu and Wenkai Xu and Jie Lu and Danica J. Sutherland},
Modern kernel-based two-sample tests have shown great success in distinguishing complex, high-dimensional distributions with appropriate learned kernels. Previous work has demonstrated that this kernel learning procedure succeeds, assuming a considerable number of observed samples from each distribution. In realistic scenarios with very limited numbers of data samples, however, it can be challenging to identify a kernel powerful enough to distinguish complex distributions. We address this issue… 

Figures and Tables from this paper

AutoML Two-Sample Test
This work uses a simple test that takes the mean discrepancy of a witness function as the test statistic and proves that minimizing a squared loss leads to a witness with optimal testing power to achieve competitive performance on a diverse distribution shift benchmark as well as on challenging two-sample testing problems.
Adversarial Attacks and Defense for Non-Parametric Two-Sample Tests
This paper systematically uncovers the failure mode of non-parametric TSTs through adversarial attacks and then pro-poses corresponding defense strategies to robustify and enable TST-agnostic attacks.
AgraSSt: Approximate Graph Stein Statistics for Interpretable Assessment of Implicit Graph Generators
A novel statistical procedure, coined AgraSSt, which can be used to determine whether a learned graph generating process is capable of generating graphs which resemble a given input graph and gives theoretical guarantees for a broad class of random graph models.
Bilateral Dependency Optimization: Defending Against Model-inversion Attacks
Experiments show that BiDO achieves the state-of-the-art defense performance for a variety of datasets, classifiers, and MI attacks while suffering a minor classification-accuracy drop compared to the well-trained classifier with no defense, which lights up a novel road to defend against MI attacks.
Multi-class Classification with Fuzzy-feature Observations: Theory and Algorithms
This paper gives the theoretical analysis of the MCIMO problem based on fuzzy Rademacher complexity, and two practical algorithms based on support vector machine and neural networks are constructed to solve the proposed new problem.


Learning Deep Kernels for Non-Parametric Two-Sample Tests
A class of kernel-based two-sample tests, which aim to determine whether two sets of samples are drawn from the same distribution, which applies both to kernels on deep features and to simpler radial basis kernels or multiple kernel learning.
An Optimal Witness Function for Two-Sample Testing
The proposed WiTS test can achieve higher test power than existing two-sample tests with optimized kernels, suggesting that learning a highor infinitedimensional representation of the data may not be necessary for two- sample testing.
Learning Kernel Tests Without Data Splitting
This work proposes an approach that enables learning the hyperparameters and testing on the full sample without data splitting, and can correctly calibrate the test in the presence of such dependency, and yield a test threshold in closed form.
Two-sample Testing Using Deep Learning
A two-sample testing procedure based on learned deep neural network representations that perform an asymptotic location test on data samples mapped onto a hidden layer that yields significant decreases in type-2 error rate compared to state-of-the-art two- Sample tests such as kernel-methods and classifier two- sample tests.
Optimal kernel choice for large-scale two-sample tests
The new kernel selection approach yields a more powerful test than earlier kernel selection heuristics, and makes the kernel selection and test procedures suited to data streams, where the observations cannot all be stored in memory.
Fast Two-Sample Testing with Analytic Representations of Probability Measures
A class of nonparametric two-sample tests with a cost linear in the sample size based on an ensemble of distances between analytic functions representing each of the distributions that give a better power/time tradeoff than competing approaches and in some cases better outright power than even the most expensive quadratic-time tests.
Learning Bounds for Open-Set Learning
This is the first study to provide a generalization bound for OSL, which is theoretically investigated by theoretically investigating the risk of the target classifier on unknown classes and proposes a novel algorithm, called auxiliary open-set risk (AOSR), to address the OSL problem.
Noise Contrastive Meta-Learning for Conditional Density Estimation using Kernel Mean Embeddings
A novel technique for meta-learning is introduced which combines neural representation and noise-contrastive estimation with the established literature of conditional mean embeddings into reproducing kernel Hilbert spaces and is validated on synthetic and real-world problems.
Adaptivity and Computation-Statistics Tradeoffs for Kernel and Distance based High Dimensional Two Sample Testing
This paper formally characterize the power of popular tests for GDA like the Maximum Mean Discrepancy with the Gaussian kernel (gMMD) and bandwidth-dependent variants of the Energy Distance with the Euclidean norm (eED) in the high-dimensional MDA regime.
A Kernel Two-Sample Test
This work proposes a framework for analyzing and comparing distributions, which is used to construct statistical tests to determine if two samples are drawn from different distributions, and presents two distribution free tests based on large deviation bounds for the maximum mean discrepancy (MMD).