• Corpus ID: 211818011

Two Routes to Scalable Credit Assignment without Weight Symmetry

@inproceedings{Kunin2020TwoRT,
  title={Two Routes to Scalable Credit Assignment without Weight Symmetry},
  author={Daniel Kunin and Aran Nayebi and Javier Sagastuy-Bre{\~n}a and Surya Ganguli and Jonathan M. Bloom and Daniel L. K. Yamins},
  booktitle={ICML},
  year={2020}
}
The neural plausibility of backpropagation has long been disputed, primarily for its use of non-local weight transport $-$ the biologically dubious requirement that one neuron instantaneously measure the synaptic weights of another. Until recently, attempts to create local learning rules that avoid weight transport have typically failed in the large-scale learning scenarios where backpropagation shines, e.g. ImageNet categorization with deep convolutional networks. Here, we investigate a… 

Figures and Tables from this paper

A Theoretical Framework for Target Propagation
TLDR
This work analyzes target propagation (TP), a popular but not yet fully understood alternative to BP, from the standpoint of mathematical optimization and shows that TP is closely related to Gauss-Newton optimization and thus substantially differs from BP.
Identifying Learning Rules From Neural Network Observables
TLDR
This work generates a large-scale dataset of learning trajectories of aggregate statistics measured in a variety of neural network architectures, loss functions, learning rule hyperparameters, and parameter initializations, and identifies the statistics of each observable that are most relevant for rule identification.
Towards Scaling Difference Target Propagation by Learning Backprop Targets
TLDR
A novel feedback weight training scheme is proposed that ensures both that DTP approximates BP and that layer-wise feedback weightTraining can be restored without sacrificing any theoretical guarantees.
How and When Random Feedback Works: A Case Study of Low-Rank Matrix Factorization
TLDR
This work investigates how and when feedback alignment (FA) works, focusing on one of the most basic problems with layered structure — low-rank matrix factorization.
On the relationship between predictive coding and backpropagation
TLDR
This manuscript reviews and extends recent work on the mathematical relationship between predictive coding and backpropagation for training feedforward artificial neural networks on supervised learning tasks and discusses a repository of functions, Torch2PC, for performing predictive coding with PyTorch neural network models.
Recurrent Connections in the Primate Ventral Visual Stream Mediate a Tradeoff Between Task Performance and Network Size During Core Object Recognition
TLDR
The results suggest the functional purpose of recurrence in the ventral pathway is to fit a high performing network in cortex, attaining computational power through temporal rather than spatial complexity.
Latent Equilibrium
TLDR
Latent Equilibrium is introduced, a new framework for inference and learning in networks of slow components which avoids these issues by harnessing the ability of biological neurons to phase-advance their output with respect to their membrane potential and enables quasi-instantaneous inference independent of network depth.
Latent Equilibrium: A unified learning theory for arbitrarily fast computation with arbitrarily slow neurons
TLDR
Latent Equilibrium is introduced, a new framework for inference and learning in networks of slow components which avoids these issues by harnessing the ability of biological neurons to phase-advance their output with respect to their membrane potential and enables quasi-instantaneous inference independent of network depth.
Benchmarking the Accuracy and Robustness of Feedback Alignment Algorithms
TLDR
BioTorch is presented, a software framework to create, train, and benchmark biologically motivated neural networks, and the performance of several feedback alignment methods proposed in the literature is investigated, thereby unveiling the importance of the forward and backward weight initialization and optimizer choice.
Credit Assignment in Neural Networks through Deep Feedback Control
TLDR
Deep Feedback Control is introduced, a new learning method that uses a feedback controller to drive a deep neural network to match a desired output target and whose control signal can be used for credit assignment, and which approximates GaussNewton optimization for a wide range of feedback connectivity patterns.
...
...

References

SHOWING 1-10 OF 35 REFERENCES
Deep Learning without Weight Transport
TLDR
Two mechanisms are described - a neural circuit called a weight mirror and a modification of an algorithm proposed by Kolen and Pollack in 1994 - both of which let the feedback path learn appropriate synaptic weights quickly and accurately even in large networks, without weight transport or complex wiring.
Algorithms for Hyper-Parameter Optimization
TLDR
This work contributes novel techniques for making response surface models P(y|x) in which many elements of hyper-parameter assignment (x) are known to be irrelevant given particular values of other elements.
Towards deep learning with segregated dendrites
TLDR
It is shown that a deep learning algorithm that utilizes multi-compartment neurons might help to understand how the neocortex optimizes cost functions, and the algorithm takes advantage of multilayer architectures to identify useful higher-order representations—the hallmark of deep learning.
Random synaptic feedback weights support error backpropagation for deep learning
TLDR
A surprisingly simple mechanism that assigns blame by multiplying errors by even random synaptic weights is presented, which can transmit teaching signals across multiple layers of neurons and performs as effectively as backpropagation on a variety of tasks.
How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation
TLDR
The motivations and theoretical justifications for this approach are laid down, along with conjectures that will have to be verified either mathematically or experimentally, including a hypothesis stating that such auto-encoder mediated target propagation could play in brains the role of credit assignment through many non-linear, noisy and discrete transformations.
Deep Learning without Weight Transport. arXiv:1904.05391 [cs, stat], April 2019
  • URL http://arxiv.org/abs/1904.05391
  • 1904
Difference Target Propagation
TLDR
It is shown that a linear correction for the imperfectness of the auto-encoders, called difference target propagation, is very effective to make target propagation actually work, leading to results comparable to back-propagation for deep networks with discrete and continuous units and denoising auto- encoders and achieving state of the art for stochastic networks.
Spike-based causal inference for weight alignment
TLDR
It is demonstrated that a simple learning rule in a spiking network can allow neurons to produce the right backward connections and thus solve the weight transport problem for biological models of learning.
Spiking allows neurons to estimate their causal effect
TLDR
It is shown how the spiking discontinuity of neurons can be a tool to estimate the causal influence of a neuron’s activity on reward and establish a link between simple learning rules and an existing causal inference method from econometrics.
Deep convolutional models improve predictions of macaque V1 responses to natural images
TLDR
Multi-layer convolutional neural networks (CNNs) set the new state of the art for predicting neural responses to natural images in primate V1 and deep features learned for object recognition are better explanations for V1 computation than all previous filter bank theories.
...
...