• Publications
  • Influence
Estimating Information Flow in Deep Neural Networks
TLDR
It is revealed that compression, i.e. reduction in I(X;T`) over the course of training, is driven by progressive geometric clustering of the representations of samples from the same class, and new evidence that compression and generalization may not be causally related is provided.
Arbitrarily Varying Wiretap Channels With Type Constrained States
TLDR
The capacity formula shows that the legitimate users effectively see an averaged main channel, while security must be ensured versus an eavesdropper with perfect channel state information, thus strengthening the previously best known single-letter upper bound by Liang et al. that has a min-max form.
Semantic-security capacity for wiretap channels of type II
TLDR
The direct proof shows that rates up to the weak-secrecy capacity of the classic WTC with a DM erasure channel to the eavesdropper are achievable and establishes the capacity of this DM wiretap EC as an upper bound for the WTC II.
Wiretap channels with random states non-causally available at the encoder
TLDR
A lower bound on the secrecy-capacity, that improves upon the previously best known result by Chen and Han Vinck, is derived based on a novel superposition coding scheme and reveals that it is at least as good as the achievable formula by Chia and El-Gamal.
Convergence of Smoothed Empirical Measures With Applications to Entropy Estimation
TLDR
Convergence of empirical measures smoothed by a Gaussian kernel is studied in terms of the Wasserstein distance, total variation (TV), Kullback-Leibler (KL) divergence, and independent variables.
Capacity of Continuous Channels with Memory via Directed Information Neural Estimator
TLDR
This work proposes a novel capacity estimation algorithm that treats the channel as a ‘black-box’, both when feedback is or is not present, and opens the door to a myriad of capacity approximation results for continuous alphabet channels that were inaccessible until now.
Gaussian-Smoothed Optimal Transport: Metric Structure and Statistical Efficiency
TLDR
This work proposes a novel Gaussian-smoothed OT (GOT) framework, that achieves the best of both worlds: preserving the 1-Wasserstein metric structure while alleviating the empirical approximation curse of dimensionality.
Asymptotic Guarantees for Generative Modeling Based on the Smooth Wasserstein Distance
TLDR
This work conducts a thorough statistical study of the minimum smooth Wasserstein estimators (MSWEs), first proving the estimator's measurability and asymptotic consistency, and characterize the limit distribution of the optimal model parameters and their associated minimal SWD.
The Information Bottleneck Problem and its Applications in Machine Learning
TLDR
The information bottleneck (IB) theory recently emerged as a bold information-theoretic paradigm for analyzing DL systems, and its recent impact on DL is surveyed.
Estimating Information Flow in Neural Networks
TLDR
An auxiliary (noisy) DNN framework is introduced, and a rigorous estimator for I(X;T) in noisy DNNs is developed, which clarifies the past observations of compression and isolates the geometric clustering of hidden representations as the true phenomenon of interest.
...
1
2
3
4
5
...