• Corpus ID: 244527239

Towards Empirical Sandwich Bounds on the Rate-Distortion Function

@article{Yang2021TowardsES,
  title={Towards Empirical Sandwich Bounds on the Rate-Distortion Function},
  author={Yibo Yang and Stephan Mandt},
  journal={ArXiv},
  year={2021},
  volume={abs/2111.12166}
}
Rate-distortion (R-D) function, a key quantity in information theory, characterizes the fundamental limit of how much a data source can be compressed subject to a fidelity criterion, by any compression algorithm. As researchers push for everimproving compression performance, establishing the R-D function of a given data source is not only of scientific interest, but also sheds light on the possible room for improving compression algorithms. Previous work on this problem relied on distributional… 

Fundamental Limits of Two-layer Autoencoders, and Achieving Them with Gradient Methods

The results characterize the minimizers of the population risk, and show that such minimizers are achieved by gradient methods; their structure is unveiled, thus leading to a concise description of the features obtained via training.

Lossy Image Compression with Quantized Hierarchical VAEs

This work redesigns ResNet VAEs using a quantization-aware posterior and prior, enabling easy quantization and entropy coding for image compression and presents a powerful andcient class of lossy image coders, outperforming previous methods on natural image (lossy) compression.

Neural Estimation of the Rate-Distortion Function for Massive Datasets

This work reformulates the rate-distortion objective, and solves the resulting functional optimization problem using neural networks, and describes a method to implement an operational lossy compression scheme with guarantees on the achievable rate and distortion.

An Introduction to Neural Data Compression

This introduction hopes to fill in the necessary background by reviewing basic coding topics such as entropy coding and rate-distortion theory, related machine learning ideas such as bits-back coding and perceptual metrics, and providing a guide through the representative works in the literature so far.

References

SHOWING 1-10 OF 77 REFERENCES

On Variational Bounds of Mutual Information

This work introduces a continuum of lower bounds that encompasses previous bounds and flexibly trades off bias and variance and demonstrates the effectiveness of these new bounds for estimation and representation learning.

Improved Variational Inference with Inverse Autoregressive Flow

A new type of normalizing flow, inverse autoregressive flow (IAF), is proposed that, in contrast to earlier published flows, scales well to high-dimensional latent spaces and significantly improves upon diagonal Gaussian approximate posteriors.

Channel-Wise Autoregressive Entropy Models for Learned Image Compression

This work introduces two enhancements, channel-conditioning and latent residual prediction, that lead to network architectures with better rate-distortion performance than existing context-adaptive models while minimizing serial processing.

Nonlinear Transform Coding

A novel variant of entropy-constrained vector quantization, based on artificial neural networks, as well as learned entropy models, is introduced to assess the empirical rate–distortion performance of nonlinear transform coding methods.

Rate Distortion Functions and Rate Distortion Function Lower Bounds for Real-World Sources

This work presents the relevant rate distortion theory and shows how this theory can be used for practical codec design and performance prediction and evaluation and illustrates the interplay between source models for rate distortion theoretic studies and the source models underlying video and speech codec design.

Masked Autoregressive Flow for Density Estimation

This work describes an approach for increasing the flexibility of an autoregressive model, based on modelling the random numbers that the model uses internally when generating data, which is called Masked Autoregressive Flow.

End-to-end Optimized Image Compression

Across an independent set of test images, it is found that the optimized method generally exhibits better rate-distortion performance than the standard JPEG and JPEG 2000 compression methods, and a dramatic improvement in visual quality is observed, supported by objective quality estimates using MS-SSIM.

On the computation of rate-distortion functions (Corresp.)

It is shown that the sequence of distributions used in that algorithm has a limit yielding a point on the R(d) curve if the reproducing alphabet is finite, and a similar but weaker result for countable reproducing alphabets is obtained.

Importance Weighted Autoencoders

The importance weighted autoencoder (IWAE), a generative model with the same architecture as the VAE, but which uses a strictly tighter log-likelihood lower bound derived from importance weighting, shows empirically that IWAEs learn richer latent space representations than VAEs, leading to improved test log- likelihood on density estimation benchmarks.

The Intrinsic Dimension of Images and Its Impact on Learning

It is found that common natural image datasets indeed have very low intrinsic dimension relative to the high number of pixels in the images, and these datasets are easier for neural networks to learn, and models solving these tasks generalize better from training to test data.
...