Corpus ID: 203837073

Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerialization

@article{Jain2020CheckmateBT,
  title={Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerialization},
  author={Paras Jain and Ajay B. Jain and Aniruddha Nrusimha and Amir Gholami and Pieter Abbeel and Kurt Keutzer and Ion Stoica and Joseph E. Gonzalez},
  journal={ArXiv},
  year={2020},
  volume={abs/1910.02653}
}
  • Paras Jain, Ajay B. Jain, +5 authors Joseph E. Gonzalez
  • Published 2020
  • Computer Science, Mathematics
  • ArXiv
  • Modern neural networks are increasingly bottlenecked by the limited capacity of on-device GPU memory. Prior work explores dropping activations as a strategy to scale to larger neural networks under memory constraints. However, these heuristics assume uniform per-layer costs and are limited to simple architectures with linear graphs, limiting their usability. In this paper, we formalize the problem of trading-off DNN training time and memory requirements as the tensor rematerialization… CONTINUE READING

    Citations

    Publications citing this paper.
    SHOWING 1-10 OF 10 CITATIONS

    Dynamic Tensor Rematerialization

    VIEW 7 EXCERPTS
    CITES BACKGROUND & METHODS
    HIGHLY INFLUENCED

    Echo: Compiler-based GPU Memory Footprint Reduction for LSTM RNN Training

    VIEW 3 EXCERPTS
    CITES METHODS & BACKGROUND

    Memory-Efficient Pipeline-Parallel DNN Training

    VIEW 1 EXCERPT
    CITES BACKGROUND

    References

    Publications referenced by this paper.
    SHOWING 1-10 OF 78 REFERENCES

    Cutting Down Training Memory by Re-fowarding

    VIEW 2 EXCERPTS

    Training Deep Nets with Sublinear Memory Cost

    VIEW 10 EXCERPTS
    HIGHLY INFLUENTIAL

    Optimal memory-aware backpropagation of deep join networks

    VIEW 1 EXCERPT

    In-place Activated BatchNorm for Memory-Optimized Training of DNNs

    VIEW 1 EXCERPT

    Gist: Efficient Data Encoding for Deep Neural Network Training

    VIEW 2 EXCERPTS