Corpus ID: 197544848

MintNet: Building Invertible Neural Networks with Masked Convolutions

@article{Song2019MintNetBI,
  title={MintNet: Building Invertible Neural Networks with Masked Convolutions},
  author={Yang Song and Chenlin Meng and Stefano Ermon},
  journal={ArXiv},
  year={2019},
  volume={abs/1907.07945}
}
We propose a new way of constructing invertible neural networks by combining simple building blocks with a novel set of composition rules. This leads to a rich set of invertible architectures, including those similar to ResNets. Inversion is achieved with a locally convergent iterative procedure that is parallelizable and very fast in practice. Additionally, the determinant of the Jacobian can be computed analytically and efficiently, enabling their generative use as flow models. To demonstrate… Expand
Understanding and mitigating exploding inverses in invertible neural networks
TLDR
This work shows that commonly-used INN architectures suffer from exploding inverses and are thus prone to becoming numerically non-invertible, and proposes a flexible and efficient regularizer for tasks where local invertibility is sufficient. Expand
HINT: Hierarchical Invertible Neural Transport for Density Estimation and Bayesian Inference
TLDR
The power of the HINT method for density estimation and Bayesian inference on a novel data set of 2D shapes in Fourier parameterization, which enables consistent visualization of samples for different dimensionalities, is demonstrated. Expand
The Convolution Exponential and Generalized Sylvester Flows
This paper introduces a new method to build linear flows, by taking the exponential of a linear transformation. This linear transformation does not need to be invertible itself, and the exponentialExpand
Semi-Supervised Learning with Normalizing Flows
TLDR
FlowGMM, an end-to-end approach to generative semi supervised learning with normalizing flows, using a latent Gaussian mixture model, is proposed, distinct in its simplicity, unified treatment of labelled and unlabelled data with an exact likelihood, interpretability, and broad applicability beyond image data. Expand
Relative Pixel Prediction For Autoregressive Image Generation
TLDR
This paper proposes to predict pixels relatively, by predicting new pixels relative to previously generated pixels (or pixels from the conditioning context, when available), and shows that this form of prediction fare favorably to its absolute counterpart when used independently, but their coordination under an unified probabilistic model yields optimal performance. Expand
Large-Capacity Image Steganography Based on Invertible Neural Networks
TLDR
The proposed large-capacity Invertible Steganography Network (ISN) takes steganography and the recovery of hidden images as a pair of inverse problems on image domain transformation, and introduces the forward and backward propagation operations of a single invertible network to leverage the image embedding and extracting problems. Expand
The Invertible U-Net for Optical-Flow-free Video Interframe Generation
TLDR
This paper is the world’s first attempt to use invertible networks instead of optical flows for video interpolation and proposes a learning method with a new consistency loss in the latent space to maintain semantic temporal consistency between frames. Expand
Argmax Flows and Multinomial Diffusion: Towards Non-Autoregressive Language Models
TLDR
This paper introduces two new classes of generative models for categorical data such as language or image segmentation: Argmax Flows and Multinomial Diffusion. Expand
Featurized Density Ratio Estimation
TLDR
This work proposes to leverage an invertible generative model to map the two distributions into a common feature space prior to estimation, bringing the densities closer together in latent space, sidestepping pathological scenarios where the learned density ratios in input space can be arbitrarily inaccurate. Expand
Learning Discrete Distributions by Dequantization
TLDR
A general framework for dequantization is presented that captures existing methods as a special case and derives two new de quantization objectives: importance-weighted (iw) dequantized and Renyi dequantification. Expand
...
1
2
3
4
...

References

SHOWING 1-10 OF 27 REFERENCES
Invertible Residual Networks
TLDR
The empirical evaluation shows that invertible ResNets perform competitively with both state-of-the-art image classifiers and flow-based generative models, something that has not been previously achieved with a single architecture. Expand
i-RevNet: Deep Invertible Networks
TLDR
The i-RevNet is built, a network that can be fully inverted up to the final projection onto the classes, i.e. no information is discarded, and linear interpolations between natural image representations are reconstructed. Expand
FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models
TLDR
This paper uses Hutchinson's trace estimator to give a scalable unbiased estimate of the log-density and demonstrates the approach on high-dimensional density estimation, image generation, and variational inference, achieving the state-of-the-art among exact likelihood methods with efficient sampling. Expand
Glow: Generative Flow with Invertible 1x1 Convolutions
TLDR
Glow, a simple type of generative flow using an invertible 1x1 convolution, is proposed, demonstrating that a generative model optimized towards the plain log-likelihood objective is capable of efficient realistic-looking synthesis and manipulation of large images. Expand
Identity Mappings in Deep Residual Networks
TLDR
The propagation formulations behind the residual building blocks suggest that the forward and backward signals can be directly propagated from one block to any other block, when using identity mappings as the skip connections and after-addition activation. Expand
MaCow: Masked Convolutional Generative Flow
TLDR
MaCow is introduced, a simple yet effective architecture of generative flow using masked convolution, which achieves significant improvements over Glow for density estimation on standard image benchmarks, considerably narrowing the gap to autoregressive models. Expand
The Reversible Residual Network: Backpropagation Without Storing Activations
TLDR
The Reversible Residual Network (RevNet) is presented, a variant of ResNets where each layer's activations can be reconstructed exactly from the next layer's, therefore, the activations for most layers need not be stored in memory during backpropagation. Expand
Emerging Convolutions for Generative Normalizing Flows
TLDR
The flexibility of d x d convolutions significantly improves the performance of generative flow models on galaxy images, CIFAR10 and ImageNet and is generalized to 1 x 1 convolutions proposed in Glow. Expand
Pixel Recurrent Neural Networks
TLDR
A deep neural network is presented that sequentially predicts the pixels in an image along the two spatial dimensions and encodes the complete set of dependencies in the image to achieve log-likelihood scores on natural images that are considerably better than the previous state of the art. Expand
Reversible Recurrent Neural Networks
TLDR
This work shows that perfectly reversible RNNs, which require no storage of the hidden activations, are fundamentally limited, and provides a scheme for storing a small number of bits in order to allow perfect reversal with forgetting. Expand
...
1
2
3
...