Monotonicity Regularization: Improved Penalties and Novel Applications to Disentangled Representation Learning and Robust Classification

  title={Monotonicity Regularization: Improved Penalties and Novel Applications to Disentangled Representation Learning and Robust Classification},
  author={Joao Monteiro and Mohamed Osama Ahmed and Hossein Hajimirsadeghi and Greg Mori},
We study settings where gradient penalties are used alongside risk minimization with the goal of ob-taining predictors satisfying different notions of monotonicity. Specifically, we present two sets of contributions. In the first part of the paper, we show that different choices of penalties define the regions of the input space where the property is observed. As such, previous methods result in models that are monotonic only in a small volume of the input space. We thus propose an approach that… 


How to Incorporate Monotonicity in Deep Networks While Preserving Flexibility
This work proposes a novel gradient-based point-wise loss function for enforcing partial monotonicity with deep neural networks, and is able to learn differentiated individual trends and produces smoother conditional curves which are important for personalized decisions, while preserving the flexibility of deep networks.
Counterexample-Guided Learning of Monotonic Neural Networks
This work develops a counterexample-guided technique to provably enforce monotonicity constraints at prediction time, and proposes a technique to usemonotonicity as an inductive bias for deep learning.
Weakly-Supervised Disentanglement Without Compromises
This work theoretically shows that only knowing how many factors have changed, but not which ones, is sufficient to learn disentangled representations, and provides practical algorithms that learn disENTangled representations from pairs of images without requiring annotation of groups, individual factors, or the number of factors that have changed.
Monotonic Kronecker-Factored Lattice
This paper proves that the function class of an ensemble of M base KFL models strictly increases as M increases up to a certain threshold, and shows that every multilinear interpolated lattice function can be expressed.
beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework
Learning an interpretable factorised representation of the independent data generative factors of the world without supervision is an important precursor for the development of artificial
Certified Monotonic Neural Networks
This work proposes to certify the monotonicity of the general piece-wise linear neural networks by solving a mixed integer linear programming problem and provides a new general approach for learning monotonic neural networks with arbitrary model structures.
Adam: A Method for Stochastic Optimization
This work introduces Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments, and provides a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework.
Deep Lattice Networks and Partial Monotonic Functions
Experiments show that six-layer monotonic deep lattice networks achieve state-of-the art performance for classification and regression with monotonicity guarantees.
MonoNet: Towards Interpretable Models by Learning Monotonic Features
It is argued that by enforcing monotonicity between features and outputs, the difficulty of interpreting a complex model stems from the existing interactions among features, and it is shown how to structurally introduce this constraint in deep learning models by adding new simple layers.
Manifold Mixup: Better Representations by Interpolating Hidden States
Manifold Mixup, a simple regularizer that encourages neural networks to predict less confidently on interpolations of hidden representations, improves strong baselines in supervised learning, robustness to single-step adversarial attacks, and test log-likelihood.