Butterfly-Net: Optimal Function Representation Based on Convolutional Neural Networks

@article{Li2020ButterflyNetOF,
  title={Butterfly-Net: Optimal Function Representation Based on Convolutional Neural Networks},
  author={Yingzhou Li and Xiuyuan Cheng and Jianfeng Lu},
  journal={ArXiv},
  year={2020},
  volume={abs/1805.07451}
}
Deep networks, especially Convolutional Neural Networks (CNNs), have been successfully applied in various areas of machine learning as well as to challenging problems in other scientific and engineering fields. This paper introduces Butterfly-net, a low-complexity CNN with structured and sparse across-channel connections, which aims at an optimal hierarchical function representation of the input signal. Theoretical analysis of the approximation power of Butterfly-net to the Fourier… 

Figures and Tables from this paper

Butterfly Transform: An Efficient FFT Based Neural Architecture Design
TLDR
It is shown that extending the butterfly operations from the FFT algorithm to a general Butterfly Transform (BFT) can be beneficial in building an efficient block structure for CNN designs, and ShuffleNet-V2+BFT outperforms state-of-the-art architecture search methods MNasNet, FBNet and MobilenetV3 in the low FLOP regime.
Wide-band butterfly network: stable and efficient inversion via multi-frequency neural networks
TLDR
An end-to-end deep learning architecture called the wide-band butterfly network (WideBNet) is introduced, which incorporates tools from computational harmonic analysis and traditional multi-scale methods to drastically reduce the number of trainable parameters to match the inherent complexity of the problem.
IAE-Net: Integral Autoencoders for Discretization-Invariant Learning
TLDR
A novel deep learning framework based on integral autoencoders (IAE-Net) for discretization invariant learning that achieves state-of-the-art performance in existing applications and creates a wide range of new applications where existing methods fail.
A multiscale neural network based on hierarchical nested bases
TLDR
A multiscale artificial neural network for high-dimensional nonlinear maps based on the idea of hierarchical nested bases in the fast multipole method and the H2-matrices to efficiently approximate discretized non linear maps arising from partial differential equations or integral equations.
Kaleidoscope: An Efficient, Learnable Representation For All Structured Linear Maps
TLDR
A family of matrices called kaleidoscope matrices (K-matrices) are introduced that provably capture any structured matrix with near-optimal space (parameter) and time (arithmetic operation) complexity that can be automatically learned within end-to-end pipelines to replace hand-crafted procedures.
SwitchNet: a neural network model for forward and inverse scattering problems
TLDR
A novel neural network architecture, SwitchNet, is proposed for solving the wave equation based inverse scattering problems via providing maps between the scatterers and the scattered field (and vice versa) by leveraging the inherent low-rank structure of the scattering problems and introducing a novel switching layer with sparse connections.
Accurate and Robust Deep Learning Framework for Solving Wave-Based Inverse Problems in the Super-Resolution Regime
TLDR
An end-to-end deep learning framework that comprehensively solves the inverse wave scattering problem across all length scales and outperforms both classical inversion and competing network architectures that specialize in oscillatory wave scattering data is proposed.
A reduced order Schwarz method for nonlinear multiscale elliptic equations based on two-layer neural networks
TLDR
A solver for multiscale fully nonlinear elliptic equations that makes use of domain decomposition, an accelerated Schwarz framework, and two-layer neural networks to approximate the boundary-to-boundary map for the subdomains, which is the key step in the Schwarz procedure.
...
1
2
...

References

SHOWING 1-10 OF 72 REFERENCES
Universality of Deep Convolutional Neural Networks
  • Ding-Xuan Zhou
  • Computer Science
    Applied and Computational Harmonic Analysis
  • 2020
Approximation Analysis of Convolutional Neural Networks ∗
TLDR
It is shown that under suitable conditions, CNNs does inherit the universal approximation property and its sample complexity can be characterized, and it is discussed concretely how the nonlinearity of T can improve the approximation power.
Optimal Approximation with Sparsely Connected Deep Neural Networks
TLDR
All function classes that are optimally approximated by a general class of representation systems---so-called affine systems---can be approximating by deep neural networks with minimal connectivity and memory requirements, and it is proved that the lower bounds are achievable for a broad family of function classes.
On the Number of Linear Regions of Deep Neural Networks
We study the complexity of functions computable by deep feedforward neural networks with piecewise linear activations in terms of the symmetries and the number of linear regions that they have. Deep
MgNet: A unified framework of multigrid and convolutional neural network
TLDR
A unified model, known as MgNet, is developed that simultaneously recovers some convolutional neural networks for image classification and multigrid methods for solving discretized partial differential equations (PDEs) and modified CNN models (with fewer weights and hyperparameters) are developed.
PDE-Net: Learning PDEs from Data
TLDR
Numerical experiments show that the PDE-Net has the potential to uncover the hidden PDE of the observed dynamics, and predict the dynamical behavior for a relatively long time, even in a noisy environment.
On the Expressive Power of Deep Learning: A Tensor Analysis
TLDR
It is proved that besides a negligible set, all functions that can be implemented by a deep network of polynomial size, require exponential size in order to be realized (or even approximated) by a shallow network.
A multiscale neural network based on hierarchical nested bases
TLDR
A multiscale artificial neural network for high-dimensional nonlinear maps based on the idea of hierarchical nested bases in the fast multipole method and the H2-matrices to efficiently approximate discretized non linear maps arising from partial differential equations or integral equations.
U-Net: Convolutional Networks for Biomedical Image Segmentation
TLDR
It is shown that such a network can be trained end-to-end from very few images and outperforms the prior best method (a sliding-window convolutional network) on the ISBI challenge for segmentation of neuronal structures in electron microscopic stacks.
Why Deep Neural Networks for Function Approximation?
TLDR
It is shown that, for a large class of piecewise smooth functions, the number of neurons needed by a shallow network to approximate a function is exponentially larger than the corresponding number of neuron needs by a deep network for a given degree of function approximation.
...
1
2
3
4
5
...