A review of automatic differentiation and its efficient implementation

@article{Margossian2019ARO,
  title={A review of automatic differentiation and its efficient implementation},
  author={Charles C. Margossian},
  journal={Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery},
  year={2019},
  volume={9}
}
  • C. Margossian
  • Published 12 November 2018
  • Computer Science, Mathematics
  • Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery
Derivatives play a critical role in computational statistics, examples being Bayesian inference using Hamiltonian Monte Carlo sampling and the training of neural networks. Automatic differentiation (AD) is a powerful tool to automate the calculation of derivatives and is preferable to more traditional methods, especially when differentiating complex algorithms and mathematical functions. The implementation of AD, however, requires some care to insure efficiency. Modern differentiation packages… Expand
Vector Forward Mode Automatic Differentiation on SIMD/SIMT architectures
TLDR
It is demonstrated that the forward mode can outperform the reverse mode for programs with tens or hundreds of directional derivatives, a number that may yet increase if current hardware trends continue. Expand
Automatic Differentiable Monte Carlo: Theory and Application
TLDR
automatic differentiable Monte Carlo paves a promising way to innovate Monte Carlo in various aspects to achieve higher accuracy and efficiency, e.g. easing or solving the sign problem of quantum many-body models through ADMC. Expand
Coarsening optimization for differentiable programming
  • Xipeng Shen, Guoqiang Zhang, +10 authors Shannon Yang
  • Computer Science
  • Proc. ACM Program. Lang.
  • 2021
TLDR
This work introduces phi-calculus, a novel method to allow symbolic reasoning and differentiation of computations that involve branches and loops and avoids "expression swell" in symbolic differentiation and balance reuse and coarsening through the design of reuse-centric segment of interest identification. Expand
Gradient and Hessian approximations in Derivative Free Optimization
This work investigates finite differences and the use of interpolation models to obtain approximations to the first and second derivatives of a function. Here, it is shown that if a particular set ofExpand
Efficient computation of nonlinear isogeometric elements using the adjoint method and algorithmic differentiation
TLDR
It turns out that using the adjoint method leads to a core-congruential formulation, which enables a clean separation between the mechanical behavior and the geometric description, which is particularly useful in CAD-integrated analysis, where mechanical properties are applied to different geometry types. Expand
Demonstration of prospective application of the dual number automatic differentiation for uncertainty propagation in neutronic calculations
TLDR
This paper solves a simplistic example of an infinite, homogeneous diffusion problem using parameters that correspond to a plate-type, Material Testing Reactor fuel assembly and demonstrates how AD can be employed in conjunction with some traditionally used sensitivity analysis and uncertainty propagation techniques. Expand
Automatic differentiation for solid mechanics
TLDR
This paper uses automatic differentiation for stating and solving solid mechanics problems, and makes use of AD for directly obtaining the residual force vector and the tangent stiffness matrix of the problem, as the gradient and the Hessian of the free energy respectively. Expand
Automatic differentiation approach for solving one-dimensional flow and heat transfer problems
  • Yuhang Niu, Yanan He, +5 authors S. Qiu
  • Computer Science
  • 2021
TLDR
It was demonstrated that the reactor system code with single-phase and five-equation two-phase flow models, which adopted the high-order discretization and the ADOO method, performed very well for one-dimensional flow and heat transfer problems. Expand
Differentiable Agent-Based Simulation for Gradient-Guided Simulation-Based Optimization
TLDR
The use of Automatic differentiation in the context of time-driven agent-based simulations is explored and it is demonstrated that the approach enables gradient-based training of neural network-controlled simulation entities embedded in the model logic. Expand
Second-order Neural Network Training Using Complex-step Directional Derivative
TLDR
This paper designs an effective Newton Krylov procedure to terminate the stochastic Krylov iteration as soon as a disturbing direction is found so that unnecessary computation can be avoided and will inspire a wide-range of new algorithms for deep learning and numerical optimization. Expand
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 87 REFERENCES
Evaluating derivatives - principles and techniques of algorithmic differentiation, Second Edition
TLDR
This second edition has been updated and expanded to cover recent developments in applications and theory, including an elegant NP completeness argument by Uwe Naumann and a brief introduction to scarcity, a generalization of sparsity. Expand
The Stan Math Library: Reverse-Mode Automatic Differentiation in C++
TLDR
The Stan Math Library is a C++, reverse-mode automatic differentiation library designed to be usable, extensive and extensible, efficient, scalable, stable, portable, and redistributable in order to facilitate the construction and utilization of such algorithms. Expand
Automatic differentiation in machine learning: a survey
TLDR
By precisely defining the main differentiation techniques and their interrelationships, this work aims to bring clarity to the usage of the terms “autodiff’, “automatic differentiation”, and “symbolic differentiation" as these are encountered more and more in machine learning settings. Expand
Algorithmic Differentiation of Implicit Functions and Optimal Values
TLDR
This paper adapts a known automatic differentiation technique for obtaining derivatives of implicitly defined functions for application to optimal value functions and develops a formulation well suited to the evaluation of first and second derivatives of optimal values. Expand
Semiautomatic Differentiation for Efficient Gradient Computations
Many large-scale computations involve a mesh and first (or sometimes higher) partial derivatives of functions of mesh elements. In principle, automatic differentiation (AD) can provide the requisiteExpand
High-Performance Derivative Computations using CoDiPack
TLDR
The new AD tool CoDiPack (Code Differentiation Package) is specifically designed for minimal memory consumption and optimal runtime, such that it can be used for the differentiation of large scale software. Expand
Fast Reverse-Mode Automatic Differentiation using Expression Templates in C++
  • R. Hogan
  • Computer Science
  • ACM Trans. Math. Softw.
  • 2014
TLDR
A fast new operator-overloading method is presented that uses the expression template programming technique in C++ to provide a compile-time representation of each mathematical expression as a computational graph that can be efficiently traversed in either direction. Expand
Algorithm 755: ADOL-C: a package for the automatic differentiation of algorithms written in C/C++
The C++ package ADOL-C described here facilitates the evaluation of first and higher derivatives of vector functions that are defined by computer programs written in C or C++. The resultingExpand
Efficient Expression Templates for Operator Overloading-based Automatic Differentiation
TLDR
Several techniques for improving the efficiency of expression templates and their implementation in the automatic differentiation package Sacado are described and their application to differentiation of a large-scale fluid dynamics simulation code is demonstrated. Expand
Achieving logarithmic growth of temporal and spatial complexity in reverse automatic differentiation
TLDR
It is shown here that, by a recursive scheme related to the multilevel differentiation approach of Volin and Ostrovskii, the growth in both temporal and spatial complexity can be limited to a fixed multiple of log(T). Expand
...
1
2
3
4
5
...