Corpus ID: 8489617

Understanding Synthetic Gradients and Decoupled Neural Interfaces

@inproceedings{Czarnecki2017UnderstandingSG,
  title={Understanding Synthetic Gradients and Decoupled Neural Interfaces},
  author={Wojciech Czarnecki and G. Swirszcz and Max Jaderberg and Simon Osindero and Oriol Vinyals and K. Kavukcuoglu},
  booktitle={ICML},
  year={2017}
}

• Wojciech Czarnecki, G. Swirszcz, +3 authors K. Kavukcuoglu
• Published in ICML 2017
• Computer Science, Mathematics

When training neural networks, the use of Synthetic Gradients (SG) allows layers or modules to be trained without update locking - without waiting for a true error gradient to be backpropagated - resulting in Decoupled Neural Interfaces (DNIs). This unlocked ability of being able to update parts of a neural network asynchronously and with only local information was demonstrated to work empirically in Jaderberg et al (2016). However, there has been very little demonstration of what changes DNIs… Expand

View PDF on arXiv

Save to Library

Create Alert

Cite

Launch Research Feed