On the Convergence of Multiplicative Update Algorithms for Nonnegative Matrix Factorization

@article{Lin2007OnTC,
  title={On the Convergence of Multiplicative Update Algorithms for Nonnegative Matrix Factorization},
  author={Chih-Jen Lin},
  journal={IEEE Transactions on Neural Networks},
  year={2007},
  volume={18},
  pages={1589-1596}
}
  • Chih-Jen Lin
  • Published 1 November 2007
  • Computer Science, Mathematics
  • IEEE Transactions on Neural Networks
Nonnegative matrix factorization (NMF) is useful to find basis information of nonnegative data. Currently, multiplicative updates are a simple and popular way to find the factorization. However, for the common NMF approach of minimizing the Euclidean distance between approximate and true values, no proof has shown that multiplicative updates converge to a stationary point of the NMF optimization problem. Stationarity is important as it is a necessary condition of a local minimum. This paper… 

Tables from this paper

A Fast Algorithm for Nonnegative Matrix Factorization and Its Convergence
TLDR
A new multiplicative update algorithm for minimizing the Euclidean distance between approximate and true values is proposed based on the optimization principle and the auxiliary function method and it is proved that this new algorithm not only converges to a stationary point, but also does faster than existing ones.
A Modified Multiplicative Update Algorithm for Euclidean Distance-Based Nonnegative Matrix Factorization and Its Global Convergence
TLDR
This paper considers NMF in which the approximation error is measured by the Euclidean distance between two matrices and proposes a modified multiplicative update algorithm in order to overcome the drawback of the original version and prove its global convergence.
A unified convergence analysis of the multiplicative update algorithm for nonnegative matrix factorization
  • Renbo Zhao, V. Tan
  • Computer Science
    2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
  • 2017
TLDR
This work provides a conceptually simple, self-contained, and unified proof for the convergence of the MU algorithm applied on NMF with a wide range of divergences and regularizations.
Global convergence of modified multiplicative updates for nonnegative matrix factorization
TLDR
Using Zangwill’s global convergence theorem, it is proved that any sequence of solutions generated by either of those modified updates has at least one convergence subsequence and the limit of any convergent subsequence is a stationary point of the corresponding optimization problem.
A Unified Convergence Analysis of the Multiplicative Update Algorithm for Regularized Nonnegative Matrix Factorization
TLDR
This work provides a conceptually simple, self-contained, and unified proof for the convergence of the MU algorithm applied on NMF with a wide range of divergences and regularizers.
Global convergence of a modified HALS algorithm for nonnegative matrix factorization
  • Takumi Kimura, Norikazu Takahashi
  • Computer Science, Mathematics
    2015 IEEE 6th International Workshop on Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP)
  • 2015
TLDR
This paper considers the HALS algorithm for the Frobenius norm-based NMF, and proves that a modified version has the global convergence property in the sense of Zangwill.
Stability analysis of multiplicative update algorithms for non-negative matrix factorization
TLDR
It is shown that Lyapunov's stability theory provides a very enlightening viewpoint on the problem of NMF multiplicative update and is proved the stability of supervised NMF and study the more difficult case of unsupervised NMF.
Stability Analysis of Multiplicative Update Algorithms and Application to Nonnegative Matrix Factorization
TLDR
It is shown that Lyapunov's stability theory provides a very enlightening viewpoint on the problem of NMF multiplicative update, and the exponential or asymptotic stability of the solutions to general optimization problems with nonnegative constraints is proved.
Multiplicative update for a class of constrained optimization problems related to NMF and its global convergence
TLDR
It is proved that a modified version of the update rule has the global convergence property in the sense of Zangwill under certain conditions and can be applied to many existing multiplicative update rules for NMF to guarantee their global convergence.
A unified global convergence analysis of multiplicative update rules for nonnegative matrix factorization
TLDR
This paper provides a sufficient condition for a general multiplicative update rule to have the global convergence property in the sense that any sequence of solutions has at least one convergent subsequence and the limit of any convergence subsequence is a stationary point of the optimization problem.
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 19 REFERENCES
Algorithms for Non-negative Matrix Factorization
TLDR
Two different multiplicative algorithms for non-negative matrix factorization are analyzed and one algorithm can be shown to minimize the conventional least squares error while the other minimizes the generalized Kullback-Leibler divergence.
Projected Gradient Methods for Nonnegative Matrix Factorization
TLDR
This letter proposes two projected gradient methods for nonnegative matrix factorization, both of which exhibit strong optimization properties and discuss efficient implementations and demonstrate that one of the proposed methods converges faster than the popular multiplicative update approach.
Optimality, computation, and interpretation of nonnegative matrix factorizations
TLDR
The theoretical Kuhn-Tucker optimality condition is described in explicit form and a number of numerical techniques, old and new, are suggested for the nonnegative matrix factorization problems.
Algorithms and applications for approximate nonnegative matrix factorization
Non-negative Matrix Factorization with Sparseness Constraints
  • P. Hoyer
  • Computer Science
    J. Mach. Learn. Res.
  • 2004
TLDR
This paper shows how explicitly incorporating the notion of 'sparseness' improves the found decompositions, and provides complete MATLAB code both for standard NMF and for an extension of this technique.
On reduced rank nonnegative matrix factorization for symmetric nonnegative matrices
Accelerating the Lee-Seung Algorithm for Nonnegative Matrix Factorization
TLDR
A variation of one of the Lee-Seung algorithms with a notably improved performance is presented and it is shown that algorithms of this type do not necessarily converge to local minima.
Positive matrix factorization: A non-negative factor model with optimal utilization of error estimates of data values†
A new variant ‘PMF’ of factor analysis is described. It is assumed that X is a matrix of observed data and σ is the known matrix of standard deviations of elements of X. Both X and σ are of
Non-negative sparse coding
  • P. Hoyer
  • Computer Science
    Proceedings of the 12th IEEE Workshop on Neural Networks for Signal Processing
  • 2002
TLDR
A simple yet efficient multiplicative algorithm for finding the optimal values of the hidden components of non-negative sparse coding and how the basis vectors can be learned from the observed data is shown.
...
1
2
...