Least Squares Support Vector Machines

An examination of least squares support vector machines (LS-SVMs) which are reformulations to standard SVMs. LS-SVMs are closely related to regularization networks and Gaussian processes but additionally emphasize and exploit primal-dual interpretations from optimization theory. The authors explain the natural links between LS-SVM classifiers and kernel Fisher discriminant analysis. Bayesian inference of LS-SVM models is discussed, together with methods for imposing sparseness and employing robust statistics. The framework is further extended towards unsupervised learning by considering PCA analysis and its kernel version as a one-class modelling problem. This leads to new primal-dual support vector machine formulations for kernel PCA and kernel CCA analysis. Furthermore, LS-SVM formulations are given for recurrent networks and control. In general, support vector machines may pose heavy computational challenges for large data sets. For this purpose, a method of fixed size LS-SVM is proposed where the estimation is done in the primal space in relation to a Nystrom sampling with active selection of support vectors. The methods are illustrated with several examples. Ch. 1. Introduction. 1.1. Multilayer perceptron neural networks. 1.2. Regression and classification. 1.3. Learning and generalization. 1.4. Principles of pattern recognition. 1.5. Dimensionality reduction methods. 1.6. Parametric versus non-parametric approaches and RBF networks. 1.7. Feedforward versus recurrent network models -- ch. 2. Support vector machines. 2.1. Maximal margin classification and linear SVMs. 2.2. Kernel trick and Mercer condition. 2.3. Nonlinear SVM classifiers. 2.4. VC theory and structural risk minimization. 2.5. SVMs for function estimation. 2.6. Modifications and extensions -- ch. 3. Basic methods of least squares support vector machines. 3.1. Least squares support vector machines for classification. 3.2. Multi-class formulations. 3.3. Link with Fisher discriminant analysis in feature space. 3.4. Solving the LS-SVM KKT system. 3.5. Least squares support vector machines for function estimation. 3.6. Links with regularization networks and Gaussian processes. 3.7. Sparseness by pruning -- ch. 4. Bayesian inference for LS-SVM models. 4.1. Bayesian inference for LS-SVM classifiers. 4.2. Bayesian inference for LS-SVM regression. 4.3. Input selection by automatic relevance determination -- ch. 5. Robustness. 5.1. Noise model assumptions and robust statistics. 5.2. Weighted LS-SVMs. 5.3. Robust cross-validation -- ch. 6. Large scale problems. 6.1. Low rank approximation methods. 6.2. Fixed size LS-SVMs. 6.3. Basis construction in the feature space. 6.4. Combining submodels -- ch. 7. LS-SVM for unsupervised learning. 7.1. Support vector machines and linear PCA analysis. 7.2. An LS-SVM approach to kernel PCA. 7.3. Links with density estimation. 7.4. Kernel CCA -- ch. 8. LS-SVM for recurrent networks and control. 8.1. Recurrent least squares support vector machines. 8.2. LS-SVMs and optimal control Annotation. This book focuses on Least Squares Support Vector Machines (LS-SVMs) which are reformulations to standard SVMs. LS-SVMs are closely related to regularization networks and Gaussian processes but additionally emphasize and exploit primal-dual interpretations from optimization theory. The authors explain the natural links between LS-SVM classifiers and kernel Fisher discriminant analysis. Bayesian inference of LS-SVM models is discussed, together with methods for imposing spareness and employing robust statistics. The framework is further extended towards unsupervised learning by considering PCA analysis and its kernel version as a one-class modelling problem. This leads to new primal-dual support vector machine formulations for kernel PCA and kernel CCA analysis. Furthermore, LS-SVM formulations are given for recurrent networks and control. In general, support vector machines may pose heavy computational challenges for large data sets. For this purpose, a method of fixed size LS-SVM is proposed where the estimation is done in the primal space in relation to a Nystrom sampling with active selection of support vectors. The methods are illustrated with several examples Na ov.: "This book focuses on Least Squares Support Vector Machines (LS-SVMs) which are reformulations to standard SVMs. LS-SVMs are closely related to regularization networks and Gaussian processes but additionally emphasize and exploit primal-dual interpretations from optimization theory. The authors explain the natural links between LS-SVM classifiers and kernel Fisher discriminant analysis. Bayesian inference of LS-SVM models is discussed, together with methods for imposing spareness and employing robust statistics. The framework is further extended towards unsupervised learning by considering PCA analysis and its kernel version as a one-class modelling problem. This leads to new primal-dual support vector machine formulations for kernel PCA and kernel CCA analysis. Furthermore, LS-SVM formulations are given for recurrent networks and control. In general, support vector machines may pose heavy computational challenges for large data sets. For this purpose, a method of fixed size LS-SVM is proposed where the estimation is done in the primal space in relation to a Nystrom sampling with active selection of support vectors. The methods are illustrated with several examples." This book focuses on Least Squares Support Vector Machines (LS-SVMs) which are reformulations to standard SVMs. LS-SVMs are closely related to regularization networks and Gaussian processes but additionally emphasize and exploit primal-dual interpretations from optimization theory. The authors explain the natural links between LS-SVM classifiers and kernel Fisher discriminant analysis. Bayesian inference of LS-SVM models is discussed, together with methods for imposing sparseness and employing robust statistics. The framework is further extended towards unsupervised learning by considering PCA analysis and its kernel version as a one-class modelling problem. This leads to new primal-dual support vector machine formulations for kernel PCA and kernel CCA analysis. Furthermore, LS-SVM formulations are given for recurrent networks and control. In general, support vector machines may pose heavy computational challenges for large data sets. For this purpose, a method of fixed size LS-SVM is proposed where the estimation is done in the primal space in relation to a Nyström sampling with active selection of support vectors. The methods are illustrated with several examples Annotation Focuses on the Least Squares Support Vector Machines (LS-SVMs) which are reformulations to standard SVMs. The authors explain the natural links between LS-SVM classifiers and kernel Fisher discriminant analysis. Bayesian inference of LS-SVM models is discussed, together with methods for imposing sparseness and employing robust statistics In the last decade many successful results have been obtained in different areas by applying neural network techniques.