Scaling the Scattering Transform: Deep Hybrid Networks

  title={Scaling the Scattering Transform: Deep Hybrid Networks},
  author={Edouard Oyallon and Eugene Belilovsky and Sergey Zagoruyko},
  journal={2017 IEEE International Conference on Computer Vision (ICCV)},
We use the scattering network as a generic and fixed initialization of the first layers of a supervised hybrid deep network. We show that early layers do not necessarily need to be learned, providing the best results to-date with pre-defined representations while being competitive with Deep CNNs. Using a shallow cascade of 1 × 1 convolutions, which encodes scattering coefficients that correspond to spatial windows of very small sizes, permits to obtain AlexNet accuracy on the imagenet… 

Figures and Tables from this paper

Scattering Networks for Hybrid Representation Learning
It is demonstrated that the early layers of CNNs do not necessarily need to be learned, and can be replaced with a scattering network instead, and using hybrid architectures, this fact is used to train hybrid GANs to generate images.
A Learnable Scatternet: Locally Invariant Convolutional Layers
This paper explores tying together the ideas from Scattering Transforms and Convolutional Neural Networks for Image Analysis by proposing a learnable ScatterNet by breaking down the scattering orders into single convolutional-like layers, and adding a learned mixing term to this layer.
Compressing the Input for CNNs with the First-Order Scattering Transform
The first-order scattering transform is studied as a candidate for reducing the signal processed by a convolutional neural network (CNN) and it is demonstrated that cascading a CNN with this representation performs on par with ImageNet classification models.
Boosting Deep Neural Networks with Geometrical Prior Knowledge: A Survey
This survey tries to give a concise overview about different approaches to incorporate geometrical prior knowledge into DNNs, and tries to connect those methods to the field of 3D object detection for autonomous driving, where they expect promising results applying those methods.
Deep Scattering Network with Max-Pooling
  • T. Ki, Youngmi Hur
  • Computer Science, Mathematics
    2021 Data Compression Conference (DCC)
  • 2021
It is shown that the scattering-maxp network shares many useful properties of the scattering network including translation invariance, and numerical experiments are conducted showing the computational advantage of the network.
Learning sparse features with lightweight ScatterNet for small sample training
Greedy Layerwise Learning Can Scale to ImageNet
This work uses 1-hidden layer learning problems to sequentially build deep networks layer by layer, which can inherit properties from shallow networks, and obtains an 11-layer network that exceeds several members of the VGG model family on ImageNet, and can train a VGG-11 model to the same accuracy as end-to-end learning.
Understanding and Simplifying Perceptual Distances
  • D. Amir, Yair Weiss
  • Computer Science, Environmental Science
    2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
  • 2021
This paper uses the tool of infinite CNNs to derive an analytical form for perceptual similarity in such CNNs, and proves that the perceptual distance between two images is equivalent to the maximum mean discrepancy (MMD) distance between local distributions of small patches in the two images.
Efficient Hybrid Network: Inducting Scattering Features
This work introduces an E-HybridNet, a novel inductive architecture that embeds scattering features into the network using Hybrid Fusion Blocks and demonstrates that the proposed design inherits the key prop-erty of prior hybrid networks - an effective generalisation in data-limited scenarios.
Resolution Learning in Deep Convolutional Networks Using Scale-Space Theory
This work uses scale-space theory to obtain a self-similar parametrization of filters and makes use of the N-Jet: a truncated Taylor series to approximate a filter by a learned combination of Gaussian derivative filters, and shows that learning σ is especially beneficial for inputs at multiple sizes.


Building a Regular Decision Boundary with Deep Networks
  • Edouard Oyallon
  • Computer Science
    2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
  • 2017
This work builds a generic architecture of Convolutional Neural Networks to discover empirical properties of neural networks and shows that the nonlinearity of a deep network does not need to be continuous, non expansive or point-wise, to achieve good performance.
Striving for Simplicity: The All Convolutional Net
It is found that max-pooling can simply be replaced by a convolutional layer with increased stride without loss in accuracy on several image recognition benchmarks.
How transferable are features in deep neural networks?
This paper quantifies the generality versus specificity of neurons in each layer of a deep convolutional neural network and reports a few surprising results, including that initializing a network with transferred features from almost any number of layers can produce a boost to generalization that lingers even after fine-tuning to the target dataset.
ImageNet classification with deep convolutional neural networks
A large, deep convolutional neural network was trained to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes and employed a recently developed regularization method called "dropout" that proved to be very effective.
Deep Residual Learning for Image Recognition
This work presents a residual learning framework to ease the training of networks that are substantially deeper than those used previously, and provides comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth.
Visualizing and Understanding Convolutional Networks
A novel visualization technique is introduced that gives insight into the function of intermediate feature layers and the operation of the classifier in large Convolutional Network models, used in a diagnostic role to find model architectures that outperform Krizhevsky et al on the ImageNet classification benchmark.
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Applied to a state-of-the-art image classification model, Batch Normalization achieves the same accuracy with 14 times fewer training steps, and beats the original model by a significant margin.
Multipath Sparse Coding Using Hierarchical Matching Pursuit
Multipath Hierarchical Matching Pursuit (M-HMP), a novel feature learning architecture that combines a collection of hierarchical sparse features for image classification to capture multiple aspects of discriminative structures, is proposed.
Intriguing properties of neural networks
It is found that there is no distinction between individual highlevel units and random linear combinations of high level units, according to various methods of unit analysis, and it is suggested that it is the space, rather than the individual units, that contains of the semantic information in the high layers of neural networks.
Learning Stable Group Invariant Representations with Convolutional Networks
It is shown that the invariance properties built by deep convolutional networks can be cast as a form of stable group invariance, enabling more abstract, powerful invariant representations.