Learning Distributions by Their Density Levels: A Paradigm for Learning without a Teacher

@article{BenDavid1997LearningDB,
  title={Learning Distributions by Their Density Levels: A Paradigm for Learning without a Teacher},
  author={Shai Ben-David and Michael Lindenbaum},
  journal={J. Comput. Syst. Sci.},
  year={1997},
  volume={55},
  pages={171-182}
}
We propose a mathematical model for learning the high-density areas of an unknown distribution from (unlabeled) random points drawn according to this distribution. While this type of a learning task has not been previously addressed in the computational learnability literature, we believe that this it a rather basic problem that appears in many practical learning scenarios. From a statistical theory standpoint, our model may be viewed as a restricted instance of the fundamental issue of… 

The Computational Complexity of Densest Region Detection

A formal learning model for this task that uses a hypothesis class as it “anti-overfitting” mechanism is introduced and it is shown that for some constants, depending on the hypothesis class, these problems are NP-hard to approximate to within these constant factors.

Pac Learning from Positive Statistical Queries ?

It is shown that k-DNF and k-decision lists are learnable in both models, i.e. with far less information than it is assumed in previously used algorithms.

PAC Learning from Positive Statistical Queries

It is shown that k-DNF and k-decision lists are learnable in both models, i.e. with far less information than it is assumed in previously used algorithms.

Learning Low Density Separators

This work proposes two natural learning paradigms and proves that, on input random samples generated i.i.d. by any distribution, they are guaranteed to converge to the optimal separator for that distribution.

Estimating the Support of a High-Dimensional Distribution

The algorithm is a natural extension of the support vector algorithm to the case of unlabeled data by carrying out sequential optimization over pairs of input patterns and providing a theoretical analysis of the statistical performance of the algorithm.

Learning Minimum Volume Sets

Support Vector Method for Novelty Detection

The algorithm is a natural extension of the support vector algorithm to the case of unlabelled data and is regularized by controlling the length of the weight vector in an associated feature space.

Accepted for NIPS ’ 99 SV Estimation of a Distribution ’ s Support

This work proposes an algorithm which approaches the problem of estimating a subset S of input space such that the probability that a test point drawn from P lies outside of S is bounded by some a priori specified 0 < 1.

A Lower Bound for Learning Distributions Generated by Probabilistic Automata

A lower bound is shown: every algorithm to learn PDFA using queries with a resonable tolerance needs a number of queries larger than (1/µ)c for every c < 1.
...

References

SHOWING 1-10 OF 25 REFERENCES

On the learnability of discrete distributions

A new model of learning probability distributions from independent draws is introduced, inspired by the popular Probably Approximately Correct (PAC) model for learning boolean functions from labeled examples, in the sense that it emphasizes efficient and approximate learning, and it studies the learnability of restricted classes of target distributions.

Probably Approximate Learning of Sets and Functions

  • B. Natarajan
  • Computer Science, Mathematics
    SIAM J. Comput.
  • 1991
The scope of the learning model is widened to include the inference of functions, and the Vapnik–Chervonenkis dimension is extended to obtain a measure called the “generalized dimension” of a class of functions.

General bounds on the number of examples needed for learning probabilistic concepts

A new method for designing learning algorithms: dynamic partitioning of the domain by use of splitting trees is introduced and it can be shown that the resulting lower bounds for learning ND are tight to within a logarithmic factor.

Efficient distribution-free learning of probabilistic concepts

  • M. KearnsR. Schapire
  • Computer Science
    Proceedings [1990] 31st Annual Symposium on Foundations of Computer Science
  • 1990
A model of machine learning in which the concept to be learned may exhibit uncertain or probabilistic behavior is investigated, and an underlying theory of learning p-concepts is developed in detail.

Localization vs. Identification of Semi-Algebraic Sets

The authors' analysis provides a mathematical ground to the intuition that localization is indeed much easier than identification and upper-bounds on the hardness of localization are established by applying a new, algebraic-geometry based, general tool for the calculation of the VC-dimension of classes of algebraically defined objects.

Bounding the Vapnik-Chervonenkis Dimension of Concept Classes Parameterized by Real Numbers

The results show that for two general kinds of concept class the V-C dimension is polynomially bounded in the number of real numbers used to define a problem instance, and that in the continuous case, as in the discrete, the real barrier to efficient learning in the Occam sense is complexity- theoretic and not information-theoretic.

Learnability and the Vapnik-Chervonenkis dimension

This paper shows that the essential condition for distribution-free learnability is finiteness of the Vapnik-Chervonenkis dimension, a simple combinatorial parameter of the class of concepts to be learned.