Corpus ID: 173990166

Discovering Neural Wirings

@inproceedings{Wortsman2019DiscoveringNW,
  title={Discovering Neural Wirings},
  author={Mitchell Wortsman and Ali Farhadi and Mohammad Rastegari},
  booktitle={NeurIPS},
  year={2019}
}
The success of neural networks has driven a shift in focus from feature engineering to architecture engineering. However, successful networks today are constructed using a small and manually defined set of building blocks. Even in methods of neural architecture search (NAS) the network connectivity patterns are largely constrained. In this work we propose a method for discovering neural wirings. We relax the typical notion of layers and instead enable channels to form connections independent of… Expand
Differentiable Dynamic Wirings for Neural Networks
  • Kun Yuan, Quanquan Li, +4 authors Ziwei Liu
A standard practice of deploying deep neural networks is to apply the same architecture to all the input instances. However, a fixed architecture may not be suitable for different data with highExpand
Mining the Weights Knowledge for Optimizing Neural Network Structures
TLDR
Inspired by how learning works in the mammalian brain, a switcher neural network is introduced that uses as inputs the weights of a task-specific neural network (called TNN for short) and mine the knowledge contained in the weights toward automatic architecture learning. Expand
Deconstructing the Structure of Sparse Neural Networks
TLDR
This work first measures performance when structure persists and weights are reset to a different random initialization, thereby extending experiments in Deconstructing Lottery Tickets, and investigates how early in training the structure emerges. Expand
Revisiting Neural Architecture Search
TLDR
This paper revisits the fundamental approach to NAS and proposes a novel approach called ReNAS that can search for the complete neural network without much human effort and is a step closer towards AutoML-nirvana. Expand
ON THE RELATIONSHIP BETWEEN TOPOLOGY AND GRADIENT PROPAGATION IN DEEP NETWORKS
  • 2020
In this paper, we address two fundamental research questions in neural architecture design: (i) How does the architecture topology impact the gradient flow during training? (ii) Can certainExpand
Dynamic Graph: Learning Instance-aware Connectivity for Neural Networks
TLDR
The Dynamic Graph Network (DG-Net) is raised, which learns the instance-aware connectivity, which creates different forward paths for different instances of the network, which allows the network to have more representation ability. Expand
How Does Topology of Neural Architectures Impact Gradient Propagation and Model Performance
TLDR
This paper forms the problem of deep learning architecture design from a network science perspective and introduces a new metric called NN-Mass to quantify how effectively information flows through a given architecture, and demonstrates that the proposed Nn-Mass is more effective than the number of parameters to characterize the gradient flow properties. Expand
What’s Hidden in a Randomly Weighted Neural Network?
TLDR
It is empirically show that as randomly weighted neural networks with fixed weights grow wider and deeper, an ``untrained subnetwork" approaches a network with learned weights in accuracy. Expand
DIFFERENTIABLE NEURAL ARCHITECTURE SEARCH
  • 2018
Differentiable neural architecture search (NAS) has gained much success in discovering more flexible and diverse cell types. Current methods couple the operations and topology during search, andExpand
Operation Embeddings for Neural Architecture Search
TLDR
This paper proposes the replacement of fixed operator encoding with learnable representations in the optimization process, which effectively captures the relations of different operations, and leads to smoother and more accurate representations of the architectures and consequently to improved performance of the end task. Expand
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 46 REFERENCES
Exploring Randomly Wired Neural Networks for Image Recognition
TLDR
The results suggest that new efforts focusing on designing better network generators may lead to new breakthroughs by exploring less constrained search spaces with more room for novel design. Expand
Neural Architecture Search with Reinforcement Learning
TLDR
This paper uses a recurrent network to generate the model descriptions of neural networks and trains this RNN with reinforcement learning to maximize the expected accuracy of the generated architectures on a validation set. Expand
Deep Expander Networks: Efficient Deep Networks from Graph Theory
TLDR
This work proposes to model connections between filters of a CNN using graphs which are simultaneously sparse and well connected, using a well-studied class of graphs from theoretical computer science that satisfies these properties known as Expander graphs. Expand
Learning Implicitly Recurrent CNNs Through Parameter Sharing
TLDR
A parameter sharing scheme, in which different layers of a convolutional neural network (CNN) are defined by a learned linear combination of parameter tensors from a global bank of templates, which yields a flexible hybridization of traditional CNNs and recurrent networks. Expand
Densely Connected Convolutional Networks
TLDR
The Dense Convolutional Network (DenseNet), which connects each layer to every other layer in a feed-forward fashion, and has several compelling advantages: they alleviate the vanishing-gradient problem, strengthen feature propagation, encourage feature reuse, and substantially reduce the number of parameters. Expand
Learning Sparse Networks Using Targeted Dropout
TLDR
Target dropout is introduced, a method for training a neural network so that it is robust to subsequent pruning, and improves upon more complicated sparsifying regularisers while being simple to implement and easy to tune. Expand
Luck Matters: Understanding Training Dynamics of Deep ReLU Networks
TLDR
Using a teacher-student setting, a novel relationship between the gradient received by hidden student nodes and the activations of teacher nodes for deep ReLU networks is discovered and it is proved that student nodes whose weights are initialized to be close to teacher nodes converge to them at a faster rate. Expand
Learning Multiple Layers of Features from Tiny Images
TLDR
It is shown how to train a multi-layer generative model that learns to extract meaningful features which resemble those found in the human visual cortex, using a novel parallelization algorithm to distribute the work among multiple machines connected on a network. Expand
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
TLDR
This work finds that dense, randomly-initialized, feed-forward networks contain subnetworks ("winning tickets") that - when trained in isolation - reach test accuracy comparable to the original network in a similar number of iterations, and articulate the "lottery ticket hypothesis". Expand
Learning Sparse Neural Networks through L0 Regularization
TLDR
A practical method for L_0 norm regularization for neural networks: pruning the network during training by encouraging weights to become exactly zero, which allows for straightforward and efficient learning of model structures with stochastic gradient descent and allows for conditional computation in a principled way. Expand
...
1
2
3
4
5
...