Sifting Common Information from Many Variables

@inproceedings{Steeg2017SiftingCI,
  title={Sifting Common Information from Many Variables},
  author={Greg Ver Steeg and Shuyang Gao and Kyle Reing and A. G. Galstyan},
  booktitle={IJCAI},
  year={2017}
}
Measuring the relationship between any pair of variables is a rich and active area of research that is central to scientific practice. In contrast, characterizing the common information among any group of variables is typically a theoretical exercise with few practical methods for high-dimensional data. A promising solution would be a multivariate generalization of the famous Wyner common information, but this approach relies on solving an apparently intractable optimization problem. We… Expand
The Information Sieve
TLDR
A new framework for unsupervised learning of representations based on a novel hierarchical decomposition of information is introduced and applied to a variety of fundamental tasks in unsuper supervised learning including independent component analysis, lossy and lossless compression, and predicting missing values in data. Expand
Algebraic properties of solutions to common information of Gaussian vectors under sparse graphical constraints
  • A. Moharrer, Shuangqing Wei
  • Mathematics, Computer Science
  • 2017 55th Annual Allerton Conference on Communication, Control, and Computing (Allerton)
  • 2017
We formulate Wyner's common information for random vectors x ∊ Rn with joint Gaussian density. We show that finding the common information of Gaussian vectors is equivalent to maximizing aExpand
Unsupervised Learning via Total Correlation Explanation
TLDR
The principle of Total Cor-relation Ex-planation (CorEx) is to learn representations of data that "explain" as much dependence in the data as possible to solve unsupervised learning problems across diverse domains including human behavior, biology, and language. Expand
Nonparanormal Information Estimation
TLDR
This work proposes estimators for mutual information when p is assumed to be a nonparanormal (a.k.a., Gaussian copula) model, a semiparametric compromise between Gaussian and nonparametric extremes, and shows these estimators strike a practical balance between robustness and scaling with dimensionality. Expand
Layered Synthesis of Latent Gaussian Trees
TLDR
It is shown that maximizing the achievable rate-region is equivalent to finding the worst case density for Bernoulli sign inputs where maximum amount of sign information is lost, and quantifies the amount of information loss due to unrecoverable sign information. Expand
PADDLE: Performance Analysis Using a Data-Driven Learning Environment
TLDR
This work proposes a unified machine learning framework, PADDLE, which is specifically designed for problems encountered during analysis of HPC data and can produce highly robust and interpretable models. Expand
Information Theoretic Study of Gaussian Graphical Models and Their Applications
TLDR
The author’s aim is to provide a “roadmap” for the development of a coherent theory of randomness in trees based on topological and algebraic perspectives. Expand

References

SHOWING 1-10 OF 68 REFERENCES
Information Bottleneck for Gaussian Variables
TLDR
A formal definition of the general continuous IB problem is given and an analytic solution for the optimal representation for the important case of multivariate Gaussian variables is obtained, in terms of the eigenvalue spectrum. Expand
The Information Sieve
TLDR
A new framework for unsupervised learning of representations based on a novel hierarchical decomposition of information is introduced and applied to a variety of fundamental tasks in unsuper supervised learning including independent component analysis, lossy and lossless compression, and predicting missing values in data. Expand
Maximally Informative Hierarchical Representations of High-Dimensional Data
TLDR
A new approach to unsupervised learning of deep representations that is both principled and practical is established and demonstrates the usefulness of the approach on both synthetic and real-world data. Expand
Discovering Structure in High-Dimensional Data Through Correlation Explanation
TLDR
It is demonstrated that Correlation Explanation automatically discovers meaningful structure for data from diverse sources including personality tests, DNA, and human language. Expand
Low Complexity Gaussian Latent Factor Models and a Blessing of Dimensionality
TLDR
This work designs a new approach to learning Gaussian latent factor models that benefits from dimensionality and relies on an unconstrained information-theoretic objective whose global optima correspond to structured latent factor generative models. Expand
A global geometric framework for nonlinear dimensionality reduction.
TLDR
An approach to solving dimensionality reduction problems that uses easily measured local metric information to learn the underlying global geometry of a data set and efficiently computes a globally optimal solution, and is guaranteed to converge asymptotically to the true structure. Expand
Representation Learning: A Review and New Perspectives
TLDR
Recent work in the area of unsupervised feature learning and deep learning is reviewed, covering advances in probabilistic models, autoencoders, manifold learning, and deep networks. Expand
NICE: Non-linear Independent Components Estimation
We propose a deep learning framework for modeling complex high-dimensional densities called Non-linear Independent Component Estimation (NICE). It is based on the idea that a good representation isExpand
Independent component analysis: algorithms and applications
TLDR
The basic theory and applications of ICA are presented, and the goal is to find a linear representation of non-Gaussian data so that the components are statistically independent, or as independent as possible. Expand
Independent component analysis, A new concept?
  • P. Comon
  • Mathematics, Computer Science
  • Signal Process.
  • 1994
TLDR
An efficient algorithm is proposed, which allows the computation of the ICA of a data matrix within a polynomial time and may actually be seen as an extension of the principal component analysis (PCA). Expand
...
1
2
3
4
5
...