Solving Ill-Conditioned and Singular Linear Systems: A Tutorial on Regularization

  title={Solving Ill-Conditioned and Singular Linear Systems: A Tutorial on Regularization},
  author={Arnold Neumaier},
  journal={SIAM Rev.},
  • A. Neumaier
  • Published 1 September 1998
  • Mathematics
  • SIAM Rev.
It is shown that the basic regularization procedures for finding meaningful approximate solutions of ill-conditioned or singular linear systems can be phrased and analyzed in terms of classical linear algebra that can be taught in any numerical analysis course. Apart from rewriting many known results in a more elementary form, we also derive a new two-parameter family of merit functions for the determination of the regularization parameter. The traditional merit functions from generalized cross… 

Tables from this paper

On the numerical solution of ill‐conditioned linear systems by regularization and iteration
  • R. Spigler
  • Mathematics
    Numer. Linear Algebra Appl.
  • 2021
We propose to reduce the (spectral) condition number of a given linear system by adding a suitable diagonal matrix to the system matrix, in particular by shifting its spectrum. Iterative procedures
Lagrangian methods for the regularization of discrete ill-posed problems
  • G. Landi
  • Mathematics, Computer Science
  • 2005
The results indicate that the proposed strategies are effective and efficient in computing good regularized solutions of ill-conditioned linear systems as well as the corresponding regularization parameters.
Numerical differentiation from a viewpoint of regularization theory
New rules are proposed for the choice of the stepsize in the finite-difference methods, and for the regularization parameter choice in numerical differentiation regularized by the iterated Tikhonov method, which are shown to be effective for the differentiation of noisy functions.
The Best Approximate Solution of the Inconsistent Linear System via a Quadratic Penalty Function
In this paper, the inconsistent linear system of $$m$$m equations in $$n$$n unknowns is formulated as a quadratic programming problem, and the best approximate solution with the minimum norm for the
On Regularization of Least Square Problems via Quadratic Constraints
  • M. Fozunbal
  • Mathematics
    2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP '07
  • 2007
To deal with the dual problem of finding the optimal Lagrange multipliers, this work introduces an approximate, positive semidefinite program that can be easily solved using the standard numerical techniques.
The regularization theory of the Krylov iterative solvers LSQR and CGLS for linear discrete ill-posed problems, part I: the simple singular value case
For the large-scale linear discrete ill-posed problem $\min\|Ax-b\|$ or $Ax=b$ with $b$ contaminated by a white noise, the Lanczos bidiagonalization based LSQR method and its mathematically
The Lagrange method for the regularization of discrete ill-posed problems
  • G. Landi
  • Computer Science, Mathematics
    Comput. Optim. Appl.
  • 2008
The results indicate that the proposed Lagrange method is effective and efficient in computing good regularized solutions of ill-conditioned linear systems and in computing the corresponding Lagrange multipliers, and the numerical experiments show that the Lag Range method is computationally convenient.
The Regularization Theory of the Krylov Iterative Solvers LSQR, CGLS, LSMR and CGME For Linear Discrete Ill-Posed Problems
For the large-scale linear discrete ill-posed problem $\min\|Ax-b\|$ or $Ax=b$ with $b$ contaminated by a white noise, Lanczos bidiagonalization based LSQR and its mathematically equivalent CGLS are


Algorithms for the regularization of ill-conditioned least squares problems
Two regularization methods for ill-conditioned least squares problems are studied from the point of view of numerical efficiency and it is shown that if they are transformed into a certain standard form, very efficient algorithms can be used for their solution.
Rank-Deficient and Discrete Ill-Posed Problems: Numerical Aspects of Linear Inversion
Preface Symbols and Acronyms 1. Setting the Stage. Problems With Ill-Conditioned Matrices Ill-Posed and Inverse Problems Prelude to Regularization Four Test Problems 2. Decompositions and Other
Analysis of Discrete Ill-Posed Problems by Means of the L-Curve
The main purpose of this paper is to advocate the use of the graph associated with Tikhonov regularization in the numerical treatment of discrete ill-posed problems, and to demonstrate several important relations between regularized solutions and the graph.
Error bounds for tikhonov regularization in hilbert scales
For an ill-posed problem Af = g we consider Tikhonov reguiariration: Minimize where is the norm in a Hilbert scale (Hs). Assuming that the norm is equivalent to the norm for some a > O and f e Hq for
Computing the Generalized Singular Value Decomposition
A new numerical method for computing the GSVD of two matrices A and B is presented, a variation on Paige''s method, which differs from previous algorithms in guaranteeing both backward stability and convergence.
A General Heuristic for Choosing the Regularization Parameter in Ill-Posed Problems
For a variety of regularization methods, including Tikhonov regularization, Landweber iteration, $\nu $-method iteration, and the method of conjugate gradients, we develop and illustrate a heuristi...
A Technique for the Numerical Solution of Certain Integral Equations of the First Kind
Here the authors will consider only nonsingular linear integral equations of the first kind, where the known functions h(x), K(x, y) and g(x) are assumed to be bounded and usually to be continuous.
Regularization of Ill-Posed Problems by Envelope Guided Conjugate Gradients
Abstract We propose a new way to iteratively solve large scale ill-posed problems by exploiting the relation between Tikhonov regularization and multiobjective optimization to obtain, iteratively,