Optimal Jacobian accumulation is NP-complete
@article{Naumann2008OptimalJA, title={Optimal Jacobian accumulation is NP-complete}, author={Uwe Naumann}, journal={Mathematical Programming}, year={2008}, volume={112}, pages={427-441} }
We show that the problem of accumulating Jacobian matrices by using a minimal number of floating-point operations is NP-complete by reduction from Ensemble Computation. The proof makes use of the fact that, deviating from the state-of-the-art assumption, algebraic dependences can exist between the local partial derivatives. It follows immediately that the same problem for directional derivatives, adjoints, and higher derivatives is NP-complete, too.
63 Citations
Derivation of Augmented Arithmetic for Computing Gradient, Hessian and Jacobian through Forward Mode AD Using Dual Numbers
- Computer Science, Mathematics
- 2009
The chain rule based differentiation arithmetic is used to find the Gradient and Hessian and Jacobian is evaluated using Gradient arithmetic by computing Gradient for components and is arranged in matrix form to give Jacobian value.
Optimization of Generalized Jacobian Chain Products without Memory Constraints
- Computer ScienceArXiv
- 2020
A dynamic programming algorithm is proposed for the minimization of the computational cost of such generalized Jacobian chain products without considering constraints on the available persistent system memory.
On lower bounds for optimal Jacobian accumulation
- Computer ScienceOptim. Methods Softw.
- 2018
A new approach for computing the lower bounds for the optimal solution of the and problems, both of which are conjectured to be NP-hard.
An Integer Programming Approach to Optimal Derivative Accumulation
- Computer Science
- 2012
This paper proposes to tackle the problem of vertex elimination with an integer programming technique, and develops an IP formulation for it, and has developed several bound-tightening and symmetry-breaking constraints to strengthen the basic IP formulation.
Algorithmic Differentiation of Numerical Methods : Tangent-Linear and Adjoint Direct Solvers for Systems of Linear Equations
- Mathematics, Computer Science
- 2012
The Algorithmic Differentiation of numerical simulation programs that contain calls to direct solvers for systems of n linear equations is considered and mathematical insight yields a reduction of the local computational complexity to O(n).
Exact Complexity Results and Polynomial-Time Algorithms for Derivative Accumulation
- Computer Science, Mathematics
- 2010
The main results concern functions derived from single-source, single-sink DAGs whose maximal paths all have length three, and derive tight, exact lower bounds for the numbers of multiplications, additions, and total arithmetic operations needed.
A Simple and Efficient Tensor Calculus
- Computer ScienceAAAI
- 2020
It is shown that using Ricci notation is not necessary for an efficient tensor calculus and developed an equally efficient method for the simpler Einstein notation, and it turns out that turning to Einstein notation enables further improvements that lead to even better efficiency.
From Semantics to Computer Science: Reversal strategies for adjoint algorithms
- Computer Science
- 2009
This paper makes a survey of the reversal strategies employed in recent tools and describes some of the more abstract formalizations used to justify these strategies.
Fast Jacobian-Vector Product for Deep Networks
- Computer ScienceArXiv
- 2021
This work proposes a novel method to quickly compute JVPs for any DN that employ Continuous Piecewise Affine (e.g., leaky-ReLU, max-pooling, maxout, etc.) nonlinearities and shows that this technique is on average 2× faster than the fastest alternative over 13 DN architectures and across various hardware.
A Simple and Efficient Tensor Calculus for Machine Learning
- Computer ScienceFundamenta Informaticae
- 2020
This paper shows that using Ricci notation is not necessary for an efficient tensor calculus and develops an equally efficient method for the simpler Einstein notation, and turns out that turning to Einstein notation enables further improvements that lead to even better efficiency.
References
SHOWING 1-10 OF 31 REFERENCES
OPTIMALITY-PRESERVING ELIMINATION OF LINEARITIES IN JACOBIAN ACCUMULATION
- Mathematics, Computer Science
- 2005
A specialized constant folding algorithm is presented to decrease the size of the combinatorial problem in order to increase the efcienc y of heuristics for its solution and it is shown that this algorithm preserves optimality in the sense that an optimal solution for the reduced problem yields an objective value no worse than that of an optimal solutions for the original problem.
On the calculation of Jacobian matrices by the Markowitz rule
- Computer Science
- 1991
This work examines this vertex elimination problem and solves it approximately, using a greedy heuristic, the resulting Markowitz scheme for Jacobian evaluation to be more efficient than column by column or row by row evaluation using the forward or the reverse mode.
Optimal accumulation of Jacobian matrices by elimination methods on the dual computational graph
- Computer Science, MathematicsMath. Program.
- 2004
Face elimination is introduced as the basic technique for accumulating Jacobian matrices by using a minimal number of arithmetic operations and its superiority over both edge and vertex elimination methods is shown.
Cheaper Jacobians by Simulated Annealing
- Computer ScienceSIAM J. Optim.
- 2002
Simulated annealing as a method for generating nearly optimal Jacobian code for combinatorial optimization problem based on vertex elimination in computational graphs.
Evaluating derivatives - principles and techniques of algorithmic differentiation, Second Edition
- BiologyFrontiers in applied mathematics
- 2000
This second edition has been updated and expanded to cover recent developments in applications and theory, including an elegant NP completeness argument by Uwe Naumann and a brief introduction to scarcity, a generalization of sparsity.
Compiling fast partial derivatives of functions given by algorithms
- Computer Science
- 1980
The system Jake described produces gradients significantly faster than numerical differencing for n > 8 and can handle algorithms Af with arbitrary flow of control.
Automatic differentiation : applications, theory, and implementations
- Computer Science
- 2006
Automatic Differentiation: A Tool for Variational Data Assimilation and Adjoint Sensitivity Analysis for Flood Modeling.
Computational differentiation : techniques, applications, and tools
- Computer Science
- 1996
This volume goes beyond the first volume published in 1991 (SIAM) in that it encompasses both the automatic transformation of computer programs as well as the methodologies for the efficient…
Solving Nonlinear Equations with Newton's Method
- Computer Science
- 1987
This chapter discusses how to get the Newton Step with Gaussian Elimination software and some of the methods used to achieve this goal.