Optimal Jacobian accumulation is NP-complete

@article{Naumann2008OptimalJA,
  title={Optimal Jacobian accumulation is NP-complete},
  author={Uwe Naumann},
  journal={Mathematical Programming},
  year={2008},
  volume={112},
  pages={427-441}
}
  • U. Naumann
  • Published 30 November 2007
  • Mathematics
  • Mathematical Programming
We show that the problem of accumulating Jacobian matrices by using a minimal number of floating-point operations is NP-complete by reduction from Ensemble Computation. The proof makes use of the fact that, deviating from the state-of-the-art assumption, algebraic dependences can exist between the local partial derivatives. It follows immediately that the same problem for directional derivatives, adjoints, and higher derivatives is NP-complete, too. 
View on Springer
cs.odu.edu
64 Citations
Background Citations
33
Methods Citations
3

64 Citations

On the Computational Complexity of the Chain Rule of Differential Calculus

TLDR
This work forms Chain Rule Differentiation and it is proved to be NP-complete, the fundamental prerequisite for computing accurate derivatives of composite functions which perform a potentially very large number of elemental function evaluations.

Derivation of Augmented Arithmetic for Computing Gradient, Hessian and Jacobian through Forward Mode AD Using Dual Numbers

TLDR
The chain rule based differentiation arithmetic is used to find the Gradient and Hessian and Jacobian is evaluated using Gradient arithmetic by computing Gradient for components and is arranged in matrix form to give Jacobian value.

Optimization of Generalized Jacobian Chain Products without Memory Constraints

TLDR
A dynamic programming algorithm is proposed for the minimization of the computational cost of such generalized Jacobian chain products without considering constraints on the available persistent system memory.

Hessian Chain Bracketing

TLDR
This work proposes a dynamic programming formulation for the solution of Hessian Accumulation over a sub-search space and yields improvements by factors of ten and higher over the state of the art based on second-order tangent and adjoint algorithmic differentiation.

On lower bounds for optimal Jacobian accumulation

TLDR
A new approach for computing the lower bounds for the optimal solution of the and problems, both of which are conjectured to be NP-hard.

A Dynamic Programming Heuristic for Dense Hessian Chain Bracketing

TLDR
Motivated by second-order parameter sensitivity analysis of surrogate numerical models obtained through training and pruning of deep neural networks this paper focusses on bracketing of dense Hessian chain products with the aim of minimizing the total number of floating-point operations to be performed.

An Integer Programming Approach to Optimal Derivative Accumulation

TLDR
This paper proposes to tackle the problem of vertex elimination with an integer programming technique, and develops an IP formulation for it, and has developed several bound-tightening and symmetry-breaking constraints to strengthen the basic IP formulation.

Algorithmic Differentiation of Numerical Methods : Tangent-Linear and Adjoint Direct Solvers for Systems of Linear Equations

TLDR
The Algorithmic Differentiation of numerical simulation programs that contain calls to direct solvers for systems of n linear equations is considered and mathematical insight yields a reduction of the local computational complexity to O(n).

A Simple and Efficient Tensor Calculus

TLDR
It is shown that using Ricci notation is not necessary for an efficient tensor calculus and developed an equally efficient method for the simpler Einstein notation, and it turns out that turning to Einstein notation enables further improvements that lead to even better efficiency.

From Semantics to Computer Science: Reversal strategies for adjoint algorithms

TLDR
This paper makes a survey of the reversal strategies employed in recent tools and describes some of the more abstract formalizations used to justify these strategies.
...

References

SHOWING 1-10 OF 31 REFERENCES

OPTIMALITY-PRESERVING ELIMINATION OF LINEARITIES IN JACOBIAN ACCUMULATION

TLDR
A specialized constant folding algorithm is presented to decrease the size of the combinatorial problem in order to increase the efcienc y of heuristics for its solution and it is shown that this algorithm preserves optimality in the sense that an optimal solution for the reduced problem yields an objective value no worse than that of an optimal solutions for the original problem.

On the calculation of Jacobian matrices by the Markowitz rule

TLDR
This work examines this vertex elimination problem and solves it approximately, using a greedy heuristic, the resulting Markowitz scheme for Jacobian evaluation to be more efficient than column by column or row by row evaluation using the forward or the reverse mode.

The Complexity of Partial Derivatives

Elimination techniques for cheap Jacobians

TLDR
It is shown that vertex elimination in general does not lead to optimal Jacobian code, and two new elimination methods are introduced that demonstrate their superiority over the vertex elimination approach.

Optimal accumulation of Jacobian matrices by elimination methods on the dual computational graph

  • U. Naumann
  • Computer Science, Mathematics
    Math. Program.
  • 2004
TLDR
Face elimination is introduced as the basic technique for accumulating Jacobian matrices by using a minimal number of arithmetic operations and its superiority over both edge and vertex elimination methods is shown.

Cheaper Jacobians by Simulated Annealing

TLDR
Simulated annealing as a method for generating nearly optimal Jacobian code for combinatorial optimization problem based on vertex elimination in computational graphs.

Evaluating derivatives - principles and techniques of algorithmic differentiation, Second Edition

TLDR
This second edition has been updated and expanded to cover recent developments in applications and theory, including an elegant NP completeness argument by Uwe Naumann and a brief introduction to scarcity, a generalization of sparsity.

Compiling fast partial derivatives of functions given by algorithms

TLDR
The system Jake described produces gradients significantly faster than numerical differencing for n > 8 and can handle algorithms Af with arbitrary flow of control.

Automatic differentiation : applications, theory, and implementations

TLDR
Automatic Differentiation: A Tool for Variational Data Assimilation and Adjoint Sensitivity Analysis for Flood Modeling.

Computational differentiation : techniques, applications, and tools

This volume goes beyond the first volume published in 1991 (SIAM) in that it encompasses both the automatic transformation of computer programs as well as the methodologies for the efficient…