# Discovering Neural Wirings

@inproceedings{Wortsman2019DiscoveringNW, title={Discovering Neural Wirings}, author={Mitchell Wortsman and Ali Farhadi and Mohammad Rastegari}, booktitle={NeurIPS}, year={2019} }

The success of neural networks has driven a shift in focus from feature engineering to architecture engineering. However, successful networks today are constructed using a small and manually defined set of building blocks. Even in methods of neural architecture search (NAS) the network connectivity patterns are largely constrained. In this work we propose a method for discovering neural wirings. We relax the typical notion of layers and instead enable channels to form connections independent of… Expand

#### Supplemental Code

#### 55 Citations

Differentiable Dynamic Wirings for Neural Networks

A standard practice of deploying deep neural networks is to apply the same architecture to all the input instances. However, a fixed architecture may not be suitable for different data with high… Expand

Mining the Weights Knowledge for Optimizing Neural Network Structures

- Computer Science
- ArXiv
- 2021

Inspired by how learning works in the mammalian brain, a switcher neural network is introduced that uses as inputs the weights of a task-specific neural network (called TNN for short) and mine the knowledge contained in the weights toward automatic architecture learning. Expand

Deconstructing the Structure of Sparse Neural Networks

- Computer Science
- ArXiv
- 2020

This work first measures performance when structure persists and weights are reset to a different random initialization, thereby extending experiments in Deconstructing Lottery Tickets, and investigates how early in training the structure emerges. Expand

Revisiting Neural Architecture Search

- Computer Science
- ArXiv
- 2020

This paper revisits the fundamental approach to NAS and proposes a novel approach called ReNAS that can search for the complete neural network without much human effort and is a step closer towards AutoML-nirvana. Expand

ON THE RELATIONSHIP BETWEEN TOPOLOGY AND GRADIENT PROPAGATION IN DEEP NETWORKS

- 2020

In this paper, we address two fundamental research questions in neural architecture design: (i) How does the architecture topology impact the gradient flow during training? (ii) Can certain… Expand

Dynamic Graph: Learning Instance-aware Connectivity for Neural Networks

- Computer Science
- ArXiv
- 2020

The Dynamic Graph Network (DG-Net) is raised, which learns the instance-aware connectivity, which creates different forward paths for different instances of the network, which allows the network to have more representation ability. Expand

How Does Topology of Neural Architectures Impact Gradient Propagation and Model Performance

- Computer Science
- 2019

This paper forms the problem of deep learning architecture design from a network science perspective and introduces a new metric called NN-Mass to quantify how effectively information flows through a given architecture, and demonstrates that the proposed Nn-Mass is more effective than the number of parameters to characterize the gradient flow properties. Expand

What’s Hidden in a Randomly Weighted Neural Network?

- Computer Science
- 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- 2020

It is empirically show that as randomly weighted neural networks with fixed weights grow wider and deeper, an ``untrained subnetwork" approaches a network with learned weights in accuracy. Expand

DIFFERENTIABLE NEURAL ARCHITECTURE SEARCH

- 2018

Differentiable neural architecture search (NAS) has gained much success in discovering more flexible and diverse cell types. Current methods couple the operations and topology during search, and… Expand

Operation Embeddings for Neural Architecture Search

- Computer Science
- ArXiv
- 2021

This paper proposes the replacement of fixed operator encoding with learnable representations in the optimization process, which effectively captures the relations of different operations, and leads to smoother and more accurate representations of the architectures and consequently to improved performance of the end task. Expand

#### References

SHOWING 1-10 OF 46 REFERENCES

Exploring Randomly Wired Neural Networks for Image Recognition

- Computer Science
- 2019 IEEE/CVF International Conference on Computer Vision (ICCV)
- 2019

The results suggest that new efforts focusing on designing better network generators may lead to new breakthroughs by exploring less constrained search spaces with more room for novel design. Expand

Neural Architecture Search with Reinforcement Learning

- Computer Science, Mathematics
- ICLR
- 2017

This paper uses a recurrent network to generate the model descriptions of neural networks and trains this RNN with reinforcement learning to maximize the expected accuracy of the generated architectures on a validation set. Expand

Deep Expander Networks: Efficient Deep Networks from Graph Theory

- Computer Science
- ECCV
- 2018

This work proposes to model connections between filters of a CNN using graphs which are simultaneously sparse and well connected, using a well-studied class of graphs from theoretical computer science that satisfies these properties known as Expander graphs. Expand

Learning Implicitly Recurrent CNNs Through Parameter Sharing

- Computer Science, Mathematics
- ICLR
- 2019

A parameter sharing scheme, in which different layers of a convolutional neural network (CNN) are defined by a learned linear combination of parameter tensors from a global bank of templates, which yields a flexible hybridization of traditional CNNs and recurrent networks. Expand

Densely Connected Convolutional Networks

- Computer Science
- 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- 2017

The Dense Convolutional Network (DenseNet), which connects each layer to every other layer in a feed-forward fashion, and has several compelling advantages: they alleviate the vanishing-gradient problem, strengthen feature propagation, encourage feature reuse, and substantially reduce the number of parameters. Expand

Learning Sparse Networks Using Targeted Dropout

- Computer Science, Mathematics
- ArXiv
- 2019

Target dropout is introduced, a method for training a neural network so that it is robust to subsequent pruning, and improves upon more complicated sparsifying regularisers while being simple to implement and easy to tune. Expand

Luck Matters: Understanding Training Dynamics of Deep ReLU Networks

- Computer Science, Mathematics
- ArXiv
- 2019

Using a teacher-student setting, a novel relationship between the gradient received by hidden student nodes and the activations of teacher nodes for deep ReLU networks is discovered and it is proved that student nodes whose weights are initialized to be close to teacher nodes converge to them at a faster rate. Expand

Learning Multiple Layers of Features from Tiny Images

- Computer Science
- 2009

It is shown how to train a multi-layer generative model that learns to extract meaningful features which resemble those found in the human visual cortex, using a novel parallelization algorithm to distribute the work among multiple machines connected on a network. Expand

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks

- Computer Science, Mathematics
- ICLR
- 2019

This work finds that dense, randomly-initialized, feed-forward networks contain subnetworks ("winning tickets") that - when trained in isolation - reach test accuracy comparable to the original network in a similar number of iterations, and articulate the "lottery ticket hypothesis". Expand

Learning Sparse Neural Networks through L0 Regularization

- Computer Science, Mathematics
- ICLR
- 2018

A practical method for L_0 norm regularization for neural networks: pruning the network during training by encouraging weights to become exactly zero, which allows for straightforward and efficient learning of model structures with stochastic gradient descent and allows for conditional computation in a principled way. Expand