• Corpus ID: 44061130

Semi-supervised Deep Kernel Learning: Regression with Unlabeled Data by Minimizing Predictive Variance

@article{Jean2018SemisupervisedDK,
  title={Semi-supervised Deep Kernel Learning: Regression with Unlabeled Data by Minimizing Predictive Variance},
  author={Neal Jean and Sang Michael Xie and Stefano Ermon},
  journal={ArXiv},
  year={2018},
  volume={abs/1805.10407}
}
Large amounts of labeled data are typically required to train deep learning models. For many real-world problems, however, acquiring additional data can be expensive or even impossible. We present semi-supervised deep kernel learning (SSDKL), a semi-supervised regression model based on minimizing predictive variance in the posterior regularization framework. SSDKL combines the hierarchical representation learning of neural networks with the probabilistic modeling capabilities of Gaussian… 
Metric-Based Semi-Supervised Regression
TLDR
The experimental results indicate that the proposed method to cope with semi-supervised regression problems achieves promising results and could capture the trend of a non-linear function and normally predict well even though this dataset comprises extreme outliers.
Deep kernels with probabilistic embeddings for small-data learning
TLDR
The proposed approach maps high-dimensional data to a probability distribution in a low dimensional subspace and then computes a kernel between these distributions to capture similarity and derive a functional gradient descent procedure for training the model.
Deep Low-Density Separation for Semi-supervised Classification
TLDR
A novel hybrid method that applies low-density separation to the embedded features of neural network-based embeddings and effectively classifies thousands of unlabeled users from a relatively small number of hand-classified examples is introduced.
Weakly-supervised Multi-output Regression via Correlated Gaussian Processes
TLDR
This paper considers multi-output regression under a weakly-supervised setting where a subset of data points from multiple groups are unlabeled and derives a modified Variational bound for fast and stable convergence in model inference and a scalable variational bound that is amenable to stochastic optimization.
Lautum Regularization for Semi-Supervised Transfer Learning
TLDR
The theory suggests that one may improve the transferability of a deep neural network by imposing a Lautum information based regularization that relates the network weights to the target data.
High-Dimensional Bayesian Optimisation with Variational Autoencoders and Deep Metric Learning
TLDR
A method combining variational autoencoders (VAEs) and deep metric learning to perform Bayesian optimisation (BO) over high-dimensional and structured input spaces is introduced, using label guidance from the blackbox function to structure the VAE latent space, facilitating the Gaussian process fit and yielding improved BO performance.
Deep Probabilistic Kernels for Sample-Efficient Learning
TLDR
Deep Probabilistic kernels are proposed which use a probabilistic neural network to map high-dimensional data to a probability distribution in a low dimensional subspace, and leverage the rich work on kernels between distributions to capture the similarity between these distributions.
A Simple yet Effective Baseline for Robust Deep Learning with Noisy Labels
TLDR
This work proposes a simple but effective baseline that is robust to noisy labels, even with severe noise, and involves a variance regularization term that implicitly penalizes the Jacobian norm of the neural network on the whole training set (including the noisy-labeled data), which encourages generalization and prevents overfitting to the corrupted labels.
Twin Neural Network Regression is a Semi-Supervised Regression Algorithm
TLDR
Semi-supervised training of twin neural network regression improves TNNR performance, which is already state of the art, significantly.
Efficient Statistical Tests: A Neural Tangent Kernel Approach
TLDR
This work proposes a shift-invariant convolutional neural tangent kernel (SCNTK) based outlier detector and two-sample tests with maximum mean discrepancy (MMD) that isO(n) in the number of samples due to using the random feature approximation.
...
1
2
3
4
...

References

SHOWING 1-10 OF 45 REFERENCES
Semi-supervised Learning by Entropy Minimization
TLDR
This framework, which motivates minimum entropy regularization, enables to incorporate unlabeled data in the standard supervised learning, and includes other approaches to the semi-supervised problem as particular or limiting cases.
Semi-Supervised Regression with Co-Training
TLDR
Experiments show that COREG can effectively exploit unlabeled data to improve regression estimates and is proposed as a co-training style semi-supervised regression algorithm.
Stochastic Variational Deep Kernel Learning
TLDR
An efficient form of stochastic variational inference is derived which leverages local kernel interpolation, inducing points, and structure exploiting algebra within this framework to enable classification, multi-task learning, additive covariance structures, and Stochastic gradient training.
Realistic Evaluation of Semi-Supervised Learning Algorithms
TLDR
This work creates a unified reimplemention and evaluation platform of various widelyused SSL techniques and finds that the performance of simple baselines which do not use unlabeled data is often underreported, that SSL methods differ in sensitivity to the amount of labeled and unlabeling data, and that performance can degrade substantially when the unlabelED dataset contains out-of-class examples.
Temporal Ensembling for Semi-Supervised Learning
TLDR
Self-ensembling is introduced, where it is shown that this ensemble prediction can be expected to be a better predictor for the unknown labels than the output of the network at the most recent training epoch, and can thus be used as a target for training.
Deep Hybrid Models: Bridging Discriminative and Generative Approaches
Most methods in machine learning are described as either discriminative or generative. The former often attain higher predictive accuracy, while the latter are more strongly regularized and can deal
Minimum variance semi-supervised boosting for multi-label classification
TLDR
The experiments show that the proposed algorithm outperforms its supervised counterpart as well as the existing information theoretic based semi-supervised methods, and its performance is steadily improving as more unlabeled data is available.
Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
TLDR
The recently proposed Temporal Ensembling has achieved state-of-the-art results in several semi-supervised learning benchmarks, but it becomes unwieldy when learning large datasets, so Mean Teacher, a method that averages model weights instead of label predictions, is proposed.
Auxiliary Deep Generative Models
TLDR
This work extends deep generative models with auxiliary variables which improves the variational approximation and proposes a model with two stochastic layers and skip connections which shows state-of-the-art performance within semi-supervised learning on MNIST, SVHN and NORB datasets.
Introduction to Semi-Supervised Learning
TLDR
This introductory book presents some popular semi-supervised learning models, including self-training, mixture models, co-training and multiview learning, graph-based methods, and semi- supervised support vector machines, and discusses their basic mathematical formulation.
...
1
2
3
4
5
...