Semi-supervised empirical risk minimization: Using unlabeled data to improve prediction

  title={Semi-supervised empirical risk minimization: Using unlabeled data to improve prediction},
  author={Oren Yuval and Saharon Rosset},
  journal={Electronic Journal of Statistics},
  • Oren Yuval, S. Rosset
  • Published 1 September 2020
  • Computer Science
  • Electronic Journal of Statistics
We present a general methodology for using unlabeled data to design semi supervised learning (SSL) variants of the Empirical Risk Minimization (ERM) learning process. Focusing on generalized linear regression, we analyze of the effectiveness of our SSL approach in improving prediction performance. The key ideas are carefully considering the null model as a competitor, and utilizing the unlabeled data to determine signal-noise combinations where SSL outperforms both supervised learning and the… 

Figures from this paper


Unlabeled data: Now it helps, now it doesn't
A finite sample analysis is developed that characterizes the value of un-labeled data and quantifies the performance improvement of SSL compared to supervised learning, and shows that there are large classes of problems for which SSL can significantly outperform supervised learning in finite sample regimes and sometimes also in terms of error convergence rates.
On the prediction loss of the lasso in the partially labeled setting
The risk bounds of the lasso estimator in the context of transductive and semi-supervised learning are revisited and it is demonstrated that the presence of a large number of unlabeled features may have significant positive impact in the situations where the restricted eigenvalue of the design matrix vanishes or is very small.
Efficient and adaptive linear regression in semi-supervised settings
A class of 'Efficient and Adaptive Semi-Supervised Estimators' (EASE) to improve estimation efficiency and establish asymPTotic results including consistency, asymptotic normality and the adaptive properties of EASE.
Semi-supervised inference: General theory and estimation of means
We propose a general semi-supervised inference framework focused on the estimation of the population mean. As usual in semi-supervised settings, there exists an unlabeled sample of covariate vectors
Semi-Supervised Linear Regression
We study a regression problem where for some part of the data we observe both the label variable ($Y$) and the predictors (${\bf X}$), while for other part of the data only the predictors are given.
From Fixed-X to Random-X Regression: Bias-Variance Decompositions, Covariance Penalties, and Prediction Error Estimation
ABSTRACT In statistical prediction, classical approaches for model selection and model evaluation based on covariance penalties are still widely used. Most of the literature on this topic is based on
Semi-supervised Learning with Deep Generative Models
It is shown that deep generative models and approximate Bayesian inference exploiting recent advances in variational methods can be used to provide significant improvements, making generative approaches highly competitive for semi-supervised learning.
A Generative Model for Semi-Supervised Learning
This paper presents a new SemiSupervised Generative Models that removes the direct dependency of data generation on label, hence overcomes this drawback and shows experiments that verifies this approach, together with comparison with existing works.
Unsupervised Semantic Aggregation and Deformable Template Matching for Semi-Supervised Learning
An Unsupervised Semantic Aggregation and Deformable Template Matching framework for SSL, which strives to improve the classification performance with few labeled data and then reduce the cost in data annotating.
Deep Learning for Industrial KPI Prediction: When Ensemble Learning Meets Semi-Supervised Data
A method called ensemble semi-supervised gated stacked AE (ES2GSAE) is proposed, in which different unlabeled datasets are used for training different submodels to ensure their diversities and can be utilized more efficiently and help enhance the model performance.