# Exchangeable Neural ODE for Set Modeling

@article{Li2020ExchangeableNO, title={Exchangeable Neural ODE for Set Modeling}, author={Yang Li and Haidong Yi and Christopher M. Bender and Siyuan Shan and Junier B. Oliva}, journal={ArXiv}, year={2020}, volume={abs/2008.02676} }

Reasoning over an instance composed of a set of vectors, like a point cloud, requires that one accounts for intra-set dependent features among elements. However, since such instances are unordered, the elements' features should remain unchanged when the input's order is permuted. This property, permutation equivariance, is a challenging constraint for most neural architectures. While recent work has proposed global pooling and attention-based solutions, these may be limited in the way that…

## 16 Citations

### SetVAE: Learning Hierarchical Composition for Generative Modeling of Set-Structured Data

- Computer Science2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- 2021

This paper proposes SetVAE, a hierarchical variational autoencoder for sets based upon attentive modules that first partition the set and project the partition back to the original cardinality, and qualitatively demonstrates that the model generalizes to unseen set sizes and learns interesting subset relations without supervision.

### Partially Observed Exchangeable Modeling

- Computer ScienceICML
- 2021

A novel framework, partially observed exchangeable modeling (POEx) that takes in a set of related partially observed instances and infers the conditional distribution for the unobserved dimensions over multiple elements, and jointly models the intrainstance and interinstance dependencies in data.

### Marginal Distribution Adaptation for Discrete Sets via Module-Oriented Divergence Minimization

- Computer ScienceICML
- 2022

Experiments on both synthetic and real-world e-commerce and EHR datasets show that the proposed framework is able to practically align a generative model to match marginal constraints under distribution shift.

### Top-N: Equivariant set and graph generation without exchangeability

- Computer ScienceArXiv
- 2021

This work addresses one-shot set and graph generation, and the parametrization of probabilistic decoders that map a vector-shaped prior to a distribution over sets or graphs, and introduces Top-n creation, a differentiable generation mechanism that uses the latent vector to select the most relevant points from a trainable reference set.

### Scalable Normalizing Flows for Permutation Invariant Densities

- Computer Science
- 2020

This work proposes an alternative way of defining permutation equivariant transformations that give closed form trace that leads not only to improvements while training, but also to better final performance.

### STEER : Simple Temporal Regularization For Neural ODEs

- Computer ScienceArXiv
- 2020

The proposed regularization technique is simple to implement, has negligible overhead and is effective across a wide variety of tasks and can significantly decrease training time and even improve performance over baseline models.

### Agent Forecasting at Flexible Horizons using ODE Flows

- Computer Science
- 2021

OMEN’s architecture embeds an assumption that marginal distributions of a given agent moving forward in time are related, allowing for an efficient representation of marginal distributions through time and allowing for reliable interpolation between prediction horizons seen in training.

### Neural Spatio-Temporal Point Processes

- Computer Science, MathematicsICLR
- 2021

We propose a new class of parameterizations for spatio-temporal point processes which leverage Neural ODEs as a computational method and enable flexible, high-fidelity models of discrete events that…

### NRTSI: Non-Recurrent Time Series Imputation for Irregularly-sampled Data

- Computer ScienceArXiv
- 2021

This work views the imputation task from the perspective of permutation equivariant modeling of sets and proposes a novel imputation model called NRTSI without any recurrent modules, which achieves state-of-the-art performance across a wide range of commonly used time series imputation benchmarks.

### NRTSI: Non-Recurrent Time Series Imputation

- Computer Science
- 2021

This work reformulate time series as permutation-equivariant sets and proposes a novel imputation model NRTSI that does not impose any recurrent structures and takes advantage of the permutation equivariant formulation.

## References

SHOWING 1-10 OF 27 REFERENCES

### BRUNO: A Deep Recurrent Model for Exchangeable Data

- Computer ScienceNeurIPS
- 2018

We present a novel model architecture which leverages deep learning tools to perform exact Bayesian inference on sets of high dimensional, complex observations. Our model is provably exchangeable,…

### Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks

- Computer ScienceICML
- 2019

This work presents an attention-based neural network module, the Set Transformer, specifically designed to model interactions among elements in the input set, and reduces the computation time of self-attention from quadratic to linear in the number of Elements in the set.

### Towards a Neural Statistician

- Computer ScienceICLR
- 2017

An extension of a variational autoencoder that can learn a method for computing representations, or statistics, of datasets in an unsupervised fashion is demonstrated that is able to learn statistics that can be used for clustering datasets, transferring generative models to new datasets, selecting representative samples of datasets and classifying previously unseen classes.

### NICE: Non-linear Independent Components Estimation

- Computer Science, MathematicsICLR
- 2015

We propose a deep learning framework for modeling complex high-dimensional densities called Non-linear Independent Component Estimation (NICE). It is based on the idea that a good representation is…

### Deep Sets

- Computer Science, MathematicsNIPS
- 2017

The main theorem characterizes the permutation invariant objective functions and provides a family of functions to which any permutation covariant objective function must belong, which enables the design of a deep network architecture that can operate on sets and which can be deployed on a variety of scenarios including both unsupervised and supervised learning tasks.

### Glow: Generative Flow with Invertible 1x1 Convolutions

- Computer ScienceNeurIPS
- 2018

Glow, a simple type of generative flow using an invertible 1x1 convolution, is proposed, demonstrating that a generative model optimized towards the plain log-likelihood objective is capable of efficient realistic-looking synthesis and manipulation of large images.

### Exchangeable Generative Models with Flow Scans

- Computer ScienceAAAI
- 2020

The proposed framework, FlowScan, combines invertible flow transformations with a sorted scan to flexibly model the data while preserving exchangeability, representing the first approach that is able to apply sequential methods to exchangeable density estimation without resorting to averaging over all possible permutations.

### Attention is All you Need

- Computer ScienceNIPS
- 2017

A new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely is proposed, which generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.

### Neural Ordinary Differential Equations

- Computer ScienceNeurIPS
- 2018

This work shows how to scalably backpropagate through any ODE solver, without access to its internal operations, which allows end-to-end training of ODEs within larger models.

### Order Matters: Sequence to sequence for sets

- Computer ScienceICLR
- 2016

An extension of the seq2seq framework that goes beyond sequences and handles input sets in a principled way is discussed and a loss is proposed which, by searching over possible orders during training, deals with the lack of structure of output sets.