Corpus ID: 219721506

Dynamic Tensor Rematerialization

@article{Kirisame2020DynamicTR,
  title={Dynamic Tensor Rematerialization},
  author={Marisa Kirisame and Steven Lyubomirsky and Altan T. Haan and Jennifer Brennan and M. He and Jared Roesch and T. Chen and Zachary Tatlock},
  journal={ArXiv},
  year={2020},
  volume={abs/2006.09616}
}
  • Marisa Kirisame, Steven Lyubomirsky, +5 authors Zachary Tatlock
  • Published 2020
  • Computer Science, Mathematics
  • ArXiv
  • Checkpointing enables training larger models by freeing intermediate activations and recomputing them on demand. Previous checkpointing techniques are difficult to generalize to dynamic models because they statically plan recomputations offline. We present Dynamic Tensor Rematerialization (DTR), a greedy online algorithm for heuristically checkpointing arbitrary models. DTR is extensible and general: it is parameterized by an eviction policy and only collects lightweight metadata on tensors and… CONTINUE READING
    1 Citations

    Figures and Tables from this paper

    Simulating Dynamic Tensor Rematerialization
    • PDF

    References

    SHOWING 1-10 OF 52 REFERENCES
    Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerialization
    • 25
    • Highly Influential
    • PDF
    A Graph Theoretic Framework of Recomputation Algorithms for Memory-Efficient Backpropagation
    • 7
    • PDF
    Optimal memory-aware backpropagation of deep join networks
    • 6
    • PDF
    Treeverse: an Implementation of Checkpointing for the Reverse or Adjoint Mode of Computational Diierentiation
    • 9
    • Highly Influential
    Efficient Rematerialization for Deep Networks
    • 6
    • Highly Influential
    • PDF
    Capuchin: Tensor-based GPU Memory Management for Deep Learning
    • 8
    • PDF
    Enabling user-driven Checkpointing strategies in Reverse-mode Automatic Differentiation
    • 12
    • PDF
    Divide-and-conquer checkpointing for arbitrary programs with no user annotation
    • 9
    • PDF