• Corpus ID: 40027675

Automatic differentiation in PyTorch

@inproceedings{Paszke2017AutomaticDI,
  title={Automatic differentiation in PyTorch},
  author={Adam Paszke and Sam Gross and Soumith Chintala and Gregory Chanan and Edward Yang and Zach DeVito and Zeming Lin and Alban Desmaison and Luca Antiga and Adam Lerer},
  year={2017}
}
In this article, we describe an automatic differentiation module of PyTorch — a library designed to enable rapid research on machine learning models. It builds upon a few projects, most notably Lua Torch, Chainer, and HIPS Autograd [4], and provides a high performance environment with easy access to automatic differentiation of models executed on different devices (CPU and GPU). To make prototyping easier, PyTorch does not follow the symbolic approach used in many other deep learning frameworks… 
gwern.net
10,068 Citations
Highly Influential Citations
1,165
Background Citations
869
Methods Citations
6,347
Results Citations
13

10,068 Citations

Citation Type

Citation Type

Torchmeta: A Meta-Learning library for PyTorch
The constant introduction of standardized benchmarks in the literature has helped accelerating the recent advances in meta-learning research. They offer a way to get a fair comparison between…
Auto-Pytorch: Multi-Fidelity MetaLearning for Efficient and Robust AutoDL
TLDR
This paper introduces Auto-PyTorch, which combines multi-fidelity optimization with portfolio construction for warmstarting and ensembling of deep neural networks (DNNs) and common baselines for tabular data to enable fully automated deep learning (AutoDL).
A Brief Introduction to Automatic Differentiation for Machine Learning
TLDR
This report describes automatic differentiation, its motivations, and different implementation approaches, and briefly describes dataflow programming as it relates to AD.
An Introduction to Automatic Differentiation for Machine Learning
TLDR
This report describes automatic differentiation, its motivations, and different implementation approaches, and briefly describes dataflow programming as it relates to AD.
PyTorch: An Imperative Style, High-Performance Deep Learning Library
TLDR
This paper details the principles that drove the implementation of PyTorch and how they are reflected in its architecture, and explains how the careful and pragmatic implementation of the key components of its runtime enables them to work together to achieve compelling performance.
fastai: A Layered API for Deep Learning
TLDR
This paper has used this library to successfully create a complete deep learning course, which was able to write more quickly than using previous approaches, and the code was more clear.
Torchbearer: A Model Fitting Library for PyTorch
We introduce torchbearer, a model fitting library for pytorch aimed at researchers working on deep learning or differentiable programming. The torchbearer library provides a high level metric and…
Pixyz: a library for developing deep generative models
TLDR
A new DGM library called Pixyz is proposed that is faster than existing probabilistic modeling languages in learning simple DGMs and can be used to implement complex DGMs in a simple and concise manner, which is difficult to do with existing libraries.
BackPACK: Packing more into backprop
TLDR
BackPACK is introduced, an efficient framework built on top of PyTorch that extends the backpropagation algorithm to extract additional information from first-and second-order derivatives to address the problem of automatic differentiation frameworks not supporting other quantities such as the variance of the mini-batch gradients.
Differentiation and Weighted Model Integration
TLDR
This note shows how to perform gradient based optimization on discrete-continuous probabilistic models, expressed as weighted model integration problems, by means of auto-differentiation, which provides a new powerful tool for inference, learning and optimization on (deep) discrete-Continuous models.
...
1
2
3
4
5
...

References

SHOWING 1-7 OF 7 REFERENCES
Modeling, Inference and Optimization With Composable Differentiable Procedures
TLDR
It is shown that early stopping and ensembling, popular tricks for avoiding overfitting, can be interpreted as variational Bayesian inference, and how to compute gradients of cross-validation loss with respect to hyperparameters of learning algorithms, with both time and memory efficiency.
DyNet: The Dynamic Neural Network Toolkit
TLDR
DyNet is a toolkit for implementing neural network models based on dynamic declaration of network structure that has an optimized C++ backend and lightweight graph representation and is designed to allow users to implement their models in a way that is idiomatic in their preferred programming language.
TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
TLDR
The TensorFlow interface and an implementation of that interface that is built at Google are described, which has been used for conducting research and for deploying machine learning systems into production across more than a dozen areas of computer science and other fields.
Evaluating derivatives - principles and techniques of algorithmic differentiation, Second Edition
TLDR
This second edition has been updated and expanded to cover recent developments in applications and theory, including an elegant NP completeness argument by Uwe Naumann and a brief introduction to scarcity, a generalization of sparsity.
Compiling fast partial derivatives of functions given by algorithms
TLDR
The system Jake described produces gradients significantly faster than numerical differencing for n > 8 and can handle algorithms Af with arbitrary flow of control.
TensorFlow: Large-scale machine learning on heterogeneous systems
  • 2015
SciPy: Open Source Scientific Tools for Python