Accelerating generalized Iterative Scaling Based on Staggered Aitken Method for on-Line Conditional Random Fields

  title={Accelerating generalized Iterative Scaling Based on Staggered Aitken Method for on-Line Conditional Random Fields},
  author={Hee-Deok Yang and Heung-Il Suk and Seong-Whan Lee},
  journal={Int. J. Wavelets Multiresolution Inf. Process.},
In this paper, a convergent method based on Generalized Iterative Scaling (GIS) with staggered Aitken acceleration is proposed to estimate the parameters for an on-line Conditional Random Field (CRF). The staggered Aitken acceleration method, which alternates between the acceleration and non-acceleration steps, ensures computational simplicity when analyzing incomplete data. The proposed method has the following advantages: (1) It can approximate parameters close to the empirical optimum in a… 

Figures and Tables from this paper



Accelerating generalized iterative scaling using componentwise extrapolations for on-line conditional random fields

In this paper, a simple and globally convergent method based on penalized generalized iterative scaling (GIS) with staggered Aitken acceleration is proposed to efficiently estimate the parameters for

On the limited memory BFGS method for large scale optimization

The numerical tests indicate that the L-BFGS method is faster than the method of Buckley and LeNir, and is better able to use additional storage to accelerate convergence, and the convergence properties are studied to prove global convergence on uniformly convex problems.

Fast Exact Multiplication by the Hessian

This work derives a technique that directly calculates Hv, where v is an arbitrary vector, and shows that this technique can be used at the heart of many iterative techniques for computing various properties of H, obviating any need to calculate the full Hessian.

Convergence analysis of gradient descent stochastic algorithms

This paper proves convergence of a sample-path based stochastic gradient-descent algorithm for optimizing expected-value performance measures in discrete event systems. The algorithm uses increasing

Introduction to stochastic search and optimization - estimation, simulation, and control

  • J. Spall
  • Computer Science
    Wiley-Interscience series in discrete mathematics and optimization
  • 2003
Introduction to Stochastic Search and Optimization: Estimation, Simulation, and Control is a graduate-level introduction to the principles, algorithms, and practical aspects of stochastic optimization, including applications drawn from engineering, statistics, and computer science.

Incremental Online Learning in High Dimensions

Locally weighted projection regression is the first truly incremental spatially localized learning method that can successfully and efficiently operate in very high-dimensional spaces.

A Stochastic Quasi-Newton Method for Online Convex Optimization

Stochastic variants of the wellknown BFGS quasi-Newton optimization method, in both full and memory-limited (LBFGS) forms, are developed for online optimization of convex functions, which asymptotically outperforms previous stochastic gradient methods for parameter estimation in conditional random fields.

Iterative solution of nonlinear equations in several variables

Convergence of Minimization Methods An Annotated List of Basic Reference Books Bibliography Author Index Subject Index.

Accelerated training of conditional random fields with stochastic gradient methods

Stochastic Meta-Descent (SMD), a stochastic gradient optimization method with gain vector adaptation, is applied to the training of Conditional Random Fields (CRFs) and the resulting optimizer converges to the same quality of solution over an order of magnitude faster than limited-memory BFGS.

Step Size Adaptation in Reproducing Kernel Hilbert Space

An online support vector machine (SVM) that uses the stochastic meta-descent (SMD) algorithm to adapt its step size automatically and applies the online SVM framework to a variety of loss functions, and in particular shows how to handle structured output spaces and achieve efficient online multiclass classification.