Learning Distributions by Their Density Levels: A Paradigm for Learning without a Teacher

@article{BenDavid1997LearningDB,
  title={Learning Distributions by Their Density Levels: A Paradigm for Learning without a Teacher},
  author={Shai Ben-David and Michael Lindenbaum},
  journal={J. Comput. Syst. Sci.},
  year={1997},
  volume={55},
  pages={171-182}
}
We propose a mathematical model for learning the high-density areas of an unknown distribution from (unlabeled) random points drawn according to this distribution. While this type of a learning task has not been previously addressed in the computational learnability literature, we believe that this it a rather basic problem that appears in many practical learning scenarios. From a statistical theory standpoint, our model may be viewed as a restricted instance of the fundamental issue of… 

The Computational Complexity of Densest Region Detection

TLDR
A formal learning model for this task that uses a hypothesis class as it “anti-overfitting” mechanism is introduced and it is shown that for some constants, depending on the hypothesis class, these problems are NP-hard to approximate to within these constant factors.

Pac Learning from Positive Statistical Queries ?

TLDR
It is shown that k-DNF and k-decision lists are learnable in both models, i.e. with far less information than it is assumed in previously used algorithms.

PAC Learning from Positive Statistical Queries

TLDR
It is shown that k-DNF and k-decision lists are learnable in both models, i.e. with far less information than it is assumed in previously used algorithms.

Learning Low Density Separators

TLDR
This work proposes two natural learning paradigms and proves that, on input random samples generated i.i.d. by any distribution, they are guaranteed to converge to the optimal separator for that distribution.

Estimating the Support of a High-Dimensional Distribution

TLDR
The algorithm is a natural extension of the support vector algorithm to the case of unlabeled data by carrying out sequential optimization over pairs of input patterns and providing a theoretical analysis of the statistical performance of the algorithm.

Learning Minimum Volume Sets

Accepted for NIPS ’ 99 SV Estimation of a Distribution ’ s Support

TLDR
This work proposes an algorithm which approaches the problem of estimating a subset S of input space such that the probability that a test point drawn from P lies outside of S is bounded by some a priori specified 0 < 1.

A Lower Bound for Learning Distributions Generated by Probabilistic Automata

TLDR
A lower bound is shown: every algorithm to learn PDFA using queries with a resonable tolerance needs a number of queries larger than (1/µ)c for every c < 1.

2 . Detecting Density Levels is a Classification Problem

TLDR
This work interprets the learning problem as a binary classification problem and compares the correspondi ng classification risk with the standard performance measure for the density level problem, and proposes a suppo rt vector machine (SVM) for anomaly detection which can easily establish universal consi stency.
...

References

SHOWING 1-10 OF 25 REFERENCES

On the learnability of discrete distributions

TLDR
A new model of learning probability distributions from independent draws is introduced, inspired by the popular Probably Approximately Correct (PAC) model for learning boolean functions from labeled examples, in the sense that it emphasizes efficient and approximate learning, and it studies the learnability of restricted classes of target distributions.

Probably Approximate Learning of Sets and Functions

  • B. Natarajan
  • Computer Science, Mathematics
    SIAM J. Comput.
  • 1991
TLDR
The scope of the learning model is widened to include the inference of functions, and the Vapnik–Chervonenkis dimension is extended to obtain a measure called the “generalized dimension” of a class of functions.

General bounds on the number of examples needed for learning probabilistic concepts

TLDR
A new method for designing learning algorithms: dynamic partitioning of the domain by use of splitting trees is introduced and it can be shown that the resulting lower bounds for learning ND are tight to within a logarithmic factor.

Efficient distribution-free learning of probabilistic concepts

  • M. KearnsR. Schapire
  • Computer Science
    Proceedings [1990] 31st Annual Symposium on Foundations of Computer Science
  • 1990
TLDR
A model of machine learning in which the concept to be learned may exhibit uncertain or probabilistic behavior is investigated, and an underlying theory of learning p-concepts is developed in detail.

Localization vs. Identification of Semi-Algebraic Sets

TLDR
The authors' analysis provides a mathematical ground to the intuition that localization is indeed much easier than identification and upper-bounds on the hardness of localization are established by applying a new, algebraic-geometry based, general tool for the calculation of the VC-dimension of classes of algebraically defined objects.

Bounding the Vapnik-Chervonenkis Dimension of Concept Classes Parameterized by Real Numbers

TLDR
The results show that for two general kinds of concept class the V-C dimension is polynomially bounded in the number of real numbers used to define a problem instance, and that in the continuous case, as in the discrete, the real barrier to efficient learning in the Occam sense is complexity- theoretic and not information-theoretic.

Learnability and the Vapnik-Chervonenkis dimension

TLDR
This paper shows that the essential condition for distribution-free learnability is finiteness of the Vapnik-Chervonenkis dimension, a simple combinatorial parameter of the class of concepts to be learned.