Unbiasing Truncated Backpropagation Through Time

@article{Tallec2017UnbiasingTB,
  title={Unbiasing Truncated Backpropagation Through Time},
  author={Corentin Tallec and Yann Ollivier},
  journal={CoRR},
  year={2017},
  volume={abs/1705.08209}
}
Truncated Backpropagation Through Time (truncated BPTT, [Jae05]) is a widespread method for learning recurrent computational graphs. Truncated BPTT keeps the computational benefits of Backpropagation Through Time (BPTT [Wer90]) while relieving the need for a complete backtrack through the whole data sequence at every step. However, truncation favors short-term dependencies: the gradient estimate of truncated BPTT is biased, so that it does not benefit from the convergence guarantees from… CONTINUE READING
Recent Discussions
This paper has been referenced on Twitter 22 times over the past 90 days. VIEW TWEETS
5 Citations
11 References
Similar Papers

Similar Papers

Loading similar papers…