Optimal Jacobian accumulation is NP-complete

@article{Naumann2008OptimalJA,
  title={Optimal Jacobian accumulation is NP-complete},
  author={Uwe Naumann},
  journal={Mathematical Programming},
  year={2008},
  volume={112},
  pages={427-441}
}
  • U. Naumann
  • Published 30 November 2007
  • Mathematics
  • Mathematical Programming
We show that the problem of accumulating Jacobian matrices by using a minimal number of floating-point operations is NP-complete by reduction from Ensemble Computation. The proof makes use of the fact that, deviating from the state-of-the-art assumption, algebraic dependences can exist between the local partial derivatives. It follows immediately that the same problem for directional derivatives, adjoints, and higher derivatives is NP-complete, too. 
View on Springer
cs.odu.edu
63 Citations
Background Citations
30
Methods Citations
3

63 Citations

Derivation of Augmented Arithmetic for Computing Gradient, Hessian and Jacobian through Forward Mode AD Using Dual Numbers
TLDR
The chain rule based differentiation arithmetic is used to find the Gradient and Hessian and Jacobian is evaluated using Gradient arithmetic by computing Gradient for components and is arranged in matrix form to give Jacobian value.
Optimization of Generalized Jacobian Chain Products without Memory Constraints
TLDR
A dynamic programming algorithm is proposed for the minimization of the computational cost of such generalized Jacobian chain products without considering constraints on the available persistent system memory.
On lower bounds for optimal Jacobian accumulation
TLDR
A new approach for computing the lower bounds for the optimal solution of the and problems, both of which are conjectured to be NP-hard.
An Integer Programming Approach to Optimal Derivative Accumulation
TLDR
This paper proposes to tackle the problem of vertex elimination with an integer programming technique, and develops an IP formulation for it, and has developed several bound-tightening and symmetry-breaking constraints to strengthen the basic IP formulation.
Algorithmic Differentiation of Numerical Methods : Tangent-Linear and Adjoint Direct Solvers for Systems of Linear Equations
TLDR
The Algorithmic Differentiation of numerical simulation programs that contain calls to direct solvers for systems of n linear equations is considered and mathematical insight yields a reduction of the local computational complexity to O(n).
Exact Complexity Results and Polynomial-Time Algorithms for Derivative Accumulation
TLDR
The main results concern functions derived from single-source, single-sink DAGs whose maximal paths all have length three, and derive tight, exact lower bounds for the numbers of multiplications, additions, and total arithmetic operations needed.
A Simple and Efficient Tensor Calculus
TLDR
It is shown that using Ricci notation is not necessary for an efficient tensor calculus and developed an equally efficient method for the simpler Einstein notation, and it turns out that turning to Einstein notation enables further improvements that lead to even better efficiency.
From Semantics to Computer Science: Reversal strategies for adjoint algorithms
TLDR
This paper makes a survey of the reversal strategies employed in recent tools and describes some of the more abstract formalizations used to justify these strategies.
Fast Jacobian-Vector Product for Deep Networks
TLDR
This work proposes a novel method to quickly compute JVPs for any DN that employ Continuous Piecewise Affine (e.g., leaky-ReLU, max-pooling, maxout, etc.) nonlinearities and shows that this technique is on average 2× faster than the fastest alternative over 13 DN architectures and across various hardware.
A Simple and Efficient Tensor Calculus for Machine Learning
TLDR
This paper shows that using Ricci notation is not necessary for an efficient tensor calculus and develops an equally efficient method for the simpler Einstein notation, and turns out that turning to Einstein notation enables further improvements that lead to even better efficiency.
...
...

References

SHOWING 1-10 OF 31 REFERENCES
OPTIMALITY-PRESERVING ELIMINATION OF LINEARITIES IN JACOBIAN ACCUMULATION
TLDR
A specialized constant folding algorithm is presented to decrease the size of the combinatorial problem in order to increase the efcienc y of heuristics for its solution and it is shown that this algorithm preserves optimality in the sense that an optimal solution for the reduced problem yields an objective value no worse than that of an optimal solutions for the original problem.
On the calculation of Jacobian matrices by the Markowitz rule
TLDR
This work examines this vertex elimination problem and solves it approximately, using a greedy heuristic, the resulting Markowitz scheme for Jacobian evaluation to be more efficient than column by column or row by row evaluation using the forward or the reverse mode.
The Complexity of Partial Derivatives
Optimal accumulation of Jacobian matrices by elimination methods on the dual computational graph
  • U. Naumann
  • Computer Science, Mathematics
    Math. Program.
  • 2004
TLDR
Face elimination is introduced as the basic technique for accumulating Jacobian matrices by using a minimal number of arithmetic operations and its superiority over both edge and vertex elimination methods is shown.
Cheaper Jacobians by Simulated Annealing
TLDR
Simulated annealing as a method for generating nearly optimal Jacobian code for combinatorial optimization problem based on vertex elimination in computational graphs.
Evaluating derivatives - principles and techniques of algorithmic differentiation, Second Edition
TLDR
This second edition has been updated and expanded to cover recent developments in applications and theory, including an elegant NP completeness argument by Uwe Naumann and a brief introduction to scarcity, a generalization of sparsity.
Compiling fast partial derivatives of functions given by algorithms
TLDR
The system Jake described produces gradients significantly faster than numerical differencing for n > 8 and can handle algorithms Af with arbitrary flow of control.
Automatic differentiation : applications, theory, and implementations
TLDR
Automatic Differentiation: A Tool for Variational Data Assimilation and Adjoint Sensitivity Analysis for Flood Modeling.
Computational differentiation : techniques, applications, and tools
This volume goes beyond the first volume published in 1991 (SIAM) in that it encompasses both the automatic transformation of computer programs as well as the methodologies for the efficient…
Solving Nonlinear Equations with Newton's Method
TLDR
This chapter discusses how to get the Newton Step with Gaussian Elimination software and some of the methods used to achieve this goal.
...
...