An Information Theoretic Approach of Designing Sparse Kernel Adaptive Filters

@article{Liu2009AnIT,
  title={An Information Theoretic Approach of Designing Sparse Kernel Adaptive Filters},
  author={Weifeng Liu and Il Memming Park and Jos{\'e} Carlos Pr{\'i}ncipe},
  journal={IEEE Transactions on Neural Networks},
  year={2009},
  volume={20},
  pages={1950-1961}
}
This paper discusses an information theoretic approach of designing sparse kernel adaptive filters. To determine useful data to be learned and remove redundant ones, a subjective information measure called surprise is introduced. Surprise captures the amount of information a datum contains which is transferable to a learning system. Based on this concept, we propose a systematic sparsification scheme, which can drastically reduce the time and space complexity without harming the performance of… 
A nonparametric information theoretic approach for change detection in time series
TLDR
An online nonparametric methodology based on the Kernel Least Mean Square (KLMS) algorithm and the surprise criterion, which is based on an information theoretic framework to detect regime change in nonstationary time series is presented.
An information theoretic kernel algorithm for robust online learning
TLDR
An information theoretic sparsification rule based on the mutual information between the system input and output to determine the update of the dictionary (support vectors) is investigated to reduce the computational cost and make the algorithm suitable for online applications.
Efficient construction of dictionaries for kernel adaptive filtering in a dynamic environment
TLDR
This paper proposes novel dictionary adaptation rules for kernel adaptive filtering that can efficiently “move” elements of the dictionary to increase the approximation performance and mainly focuses on a nonstationary system, which can yield the increase of the Dictionary size.
Self-organizing kernel adaptive filtering
TLDR
A model-selection strategy based on minimum description length (MDL) that keeps the kernel least-mean-square (KLMS) model tuned to the complexity of the input data, achieving a good compromise between system accuracy and computational complexity without a priori knowledge.
Study of Set-Membership Adaptive Kernel Algorithms
TLDR
These algorithms use an adaptive step-size to accelerate the learning and can offer an excellent tradeoff between convergence speed and steady state, which allows them to solve nonlinear filtering and estimation problems with a large number of parameters without requiring a large computational cost.
Explicit Recursive and Adaptive Filtering in Reproducing Kernel Hilbert Spaces
TLDR
Experiments in chaotic and electroencephalographic time series prediction, complex nonlinear system identification, and adaptive antenna array processing demonstrate the potential of the approach for scenarios where recursivity and nonlinearity have to be readily combined.
Combined Sparse Regularization for Nonlinear Adaptive Filters
TLDR
An adaptive combined scheme based on a block-based approach involving two nonlinear filters with different regularization that allows to achieve always superior performance than individual rules is introduced.
Online Dictionary Learning for Kernel LMS
TLDR
This theoretical analysis highlights the need for updating the dictionary in an online way, by discarding the obsolete elements and adding appropriate ones, and introduces a kernel least-mean-square algorithm with ℓ1-norm regularization to automatically perform this task.
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 47 REFERENCES
Online Prediction of Time Series Data With Kernels
TLDR
This paper investigates a new model reduction criterion that makes computationally demanding sparsification procedures unnecessary and incorporates the coherence criterion into a new kernel-based affine projection algorithm for time series prediction.
The kernel recursive least-squares algorithm
TLDR
A nonlinear version of the recursive least squares (RLS) algorithm that uses a sequential sparsification process that admits into the kernel representation a new input sample only if its feature space image cannot be sufficiently well approximated by combining the images of previously admitted samples.
Sparse On-Line Gaussian Processes
TLDR
An approach for sparse representations of gaussian process (GP) models (which are Bayesian types of kernel machines) in order to overcome their limitations for large data sets is developed based on a combination of a Bayesian on-line algorithm and a sequential construction of a relevant subsample of data that fully specifies the prediction of the GP model.
Sparse Online Gaussian Processes
TLDR
This work develops an approach for sparse representations of Gaussian Process (GP) models (which are Bayesian types of kernel machines) in order to overcome their limitations for large data sets by using an appealing parametrisation and projection techniques that use the RKHS norm.
Kernel Affine Projection Algorithms
TLDR
KAPA inherits the simplicity and online nature of KLMS while reducing its gradient noise, boosting performance and provides a unifying model for several neural network techniques, including kernel least-mean-square algorithms, kernel adaline, sliding-window kernel recursive-least squares, and regularization networks.
Incremental Gaussian Processes
In this paper, we consider Tipping's relevance vector machine (RVM) [1] and formalize an incremental training strategy as a variant of the expectation-maximization (EM) algorithm that we call
Sliding Window Generalized Kernel Affine Projection Algorithm Using Projection Mappings
TLDR
Another sparsification method for the APSM approach to the online classification task is presented by generating a sequence of linear subspaces in a reproducing kernel Hilbert space (RKHS) to cope with the inherent memory limitations of online systems and to embed tracking capabilities to the design.
Nonlinear prediction of chaotic time series using support vector machines
  • Sayan Mukherjee, E. Osuna, F. Girosi
  • Computer Science
    Neural Networks for Signal Processing VII. Proceedings of the 1997 IEEE Signal Processing Society Workshop
  • 1997
TLDR
The SVM is implemented and tested on a database of chaotic time series previously used to compare the performances of different approximation techniques, including polynomial and rational approximation, localPolynomial techniques, radial basis functions, and neural networks; the SVM performs better than the other approaches.
Online learning with kernels
TLDR
This paper considers online learning in a reproducing kernel Hilbert space, and allows the exploitation of the kernel trick in an online setting, and examines the value of large margins for classification in the online setting with a drifting target.
Fast Forward Selection to Speed Up Sparse Gaussian Process Regression
TLDR
A method for the sparse greedy approximation of Bayesian Gaussian process regression, featuring a novel heuristic for very fast forward selection, which leads to a sufficiently stable approximation of the log marginal likelihood of the training data, which can be optimised to adjust a large number of hyperparameters automatically.
...
1
2
3
4
5
...