Non-Parametric System Identification

Presenting a thorough overview of the theoretical foundations of non-parametric system identification for nonlinear block-oriented systems, this books shows that non-parametric regression can be successfully applied to system identification, and it highlights the achievements in doing so. With emphasis on Hammerstein, Wiener systems, and their multidimensional extensions, the authors show how to identify nonlinear subsystems and their characteristics when limited information exists. Algorithms using trigonometric, Legendre, Laguerre, and Hermite series are investigated, and the kernel algorithm, its semirecursive versions, and fully recursive modifications are covered. The theories of modern non-parametric regression, approximation, and orthogonal expansions, along with new approaches to system identification (including semiparametric identification), are provided. Detailed information about all tools used is provided in the appendices. This book is for researchers and practitioners in systems theory, signal processing, and communications and will appeal to researchers in fields like mechanics, economics, and biology, where experimental data are used to obtain models of systems. Half-title 3 Title 5 Copyright 6 Contents 7 Dedication 10 Preface 11 1 Introduction 13 2 Discrete-time Hammerstein systems 15 2.1 The system 15 2.2 Nonlinear subsystem 16 2.2.1 The problem and the motivation for algorithms 16 2.2.2 Simulation example 19 2.3 Dynamic subsystem identification 20 2.4 Bibliographic notes 21 3 Kernel algorithms 23 3.1 Motivation 23 3.2 Consistency 25 3.3 Applicable kernels 26 3.4 Convergence rate 28 3.5 The mean-squared error 33 3.6 Simulation example 33 3.7 Lemmas and proofs 36 3.7.1 Lemmas 36 3.7.2 Proofs 37 3.8 Bibliographic notes 41 4 Semirecursive kernel algorithms 42 4.1 Introduction 42 4.2 Consistency and convergence rate 43 4.3 Simulation example 46 4.4 Proofs and lemmas 47 4.4.1 Lemmas 47 4.4.2 Proofs 49 4.5 Bibliographic notes 55 5 Recursive kernel algorithms 56 5.1 Introduction 56 5.2 Relation to stochastic approximation 56 5.3 Consistency and convergence rate 58 5.4 Simulation example 61 5.5 Auxiliary results, lemmas, and proofs 63 5.5.1 Auxiliary results 63 5.5.2 Lemmas 65 5.5.3 Proofs 70 5.6 Bibliographic notes 70 6 Orthogonal series algorithms 71 6.1 Introduction 71 6.2 Fourier series estimate 73 6.3 Legendre series estimate 76 6.4 Laguerre series estimate 78 6.5 Hermite series estimate 80 6.6 Wavelet estimate 81 6.7 Local and global errors 82 6.8 Simulation example 83 6.9 Lemmas and proofs 84 6.9.1 Lemmas 84 6.9.2 Proofs 85 6.10 Bibliographic notes 90 7 Algorithms with ordered observations 92 7.1 Introduction 92 7.2 Kernel estimates 93 7.2.1 Motivation and estimates 93 7.2.2 Consistency and convergence rate 94 7.2.3 Simulation example 96 7.3 Orthogonal series estimates 97 7.3.1 Motivation 97 7.3.2 Fourier series estimate 98 7.3.3 Legendre series estimate 100 7.4 Lemmas and proofs 101 7.4.1 Lemmas 101 7.4.2 Proofs 104 7.5 Bibliographic notes 111 8 Continuous-time Hammerstein systems 113 8.1 Identification problem 113 8.1.1 Nonlinear subsystem identification 114 8.1.2 Dynamic subsystem identification 114 8.2 Kernel algorithm 115 8.3 Orthogonal series algorithms 118 8.4 Lemmas and proofs 120 8.4.1 Lemmas 120 8.4.2 Proofs 121 8.5 Bibliographic notes 124 9 Discrete-time Wiener systems 125 9.1 The system 125 9.2 Nonlinear subsystem 126 9.2.1 The problem and the motivation for algorithms 126 9.2.2 Possible generalizations 128 9.2.3 Monotonicity-preserving algorithms 129 9.3 Dynamic subsystem identification 131 9.3.1 The motivation 131 9.3.2 The algorithm 132 9.4 Lemmas 133 9.5 Bibliographic notes 134 10 Kernel and orthogonal series algorithms 135 10.1 Kernel algorithms 135 10.1.1 Introduction 135 10.1.2 Differentiable characteristic 136 10.1.3 Lipschitz characteristic 137 10.1.4 Convergence rate 138 10.2 Orthogonal series algorithms 138 10.2.1 Introduction 138 10.2.2 Fourier series algorithm 138 10.2.3 Legendre series algorithm 140 10.2.4 Hermite series algorithm 140 10.3 Simulation example 141 10.4 Lemmas and proofs 142 10.4.1 Lemmas 142 10.4.2 Proofs 146 10.5 Bibliographic notes 154 11 Continuous-time Wiener system 155 11.1 Identification problem 155 11.2 Nonlinear subsystem 156 11.3 Dynamic subsystem 158 11.4 Lemmas 158 11.5 Bibliographic notes 160 12 Other block-oriented nonlinear systems 161 12.1 Series-parallel, block-oriented systems 161 12.1.1 Parallel nonlinear system 161 12.1.2 Series-parallel models with nuisance characteristics 169 12.1.3 Parallel-series models 172 12.1.4 Generalized nonlinear block-oriented models 174 12.2 Block-oriented systems with nonlinear dynamics 185 12.2.1 Nonlinear models 187 12.2.2 Identification algorithms: Nonlinear system identification 191 12.2.3 Identification algorithms: Linear system identification 198 12.2.4 Identification algorithms: the Gaussian input signal 203 12.2.5 Sandwich systems with a Gaussian input signal 217 12.2.6 Convergence of identification algorithms 221 12.3 Concluding remarks 230 12.4 Bibliographical notes 232 13 Multivariate nonlinear block-oriented systems 234 13.1 Multivariate nonparametric regression 234 13.2 Additive modeling and regression analysis 240 13.2.1 Approximation by additive functions 240 13.2.2 Additive regression 245 13.3 Multivariate systems 254 13.4 Concluding remarks 260 13.5 Bibliographic notes 260 14 Semiparametric identification 262 14.1 Introduction 262 14.2 Semiparametric models 264 14.2.1 Semiparametric Hammerstein models 264 14.2.2 Semiparametric Wiener models 266 14.3 Statistical inference for semiparametric models 267 14.3.1 Consistency of semiparametric estimates 271 14.4 Statistical inference for semiparametric Wiener models 276 14.4.1 Identification algorithms 279 14.4.2 Convergence analysis 284 14.4.3 Simulation examples 290 14.4.4 Extensions 294 14.5 Statistical inference for semiparametric Hammerstein models 298 14.6 Statistical inference for semiparametric parallel models 299 14.7 Direct estimators for semiparametric systems 302 14.7.1 Average derivative estimation 302 14.7.2 The average derivative estimate for the semiparametric Wiener model 306 14.7.3 The average derivative estimate for the additive Wiener model 312 14.7.4 The average derivative estimate for semiparametric multivariate Hammerstein models 316 14.7.5 The average derivative estimate for semiparametric multivariate parallel models 319 14.8 Concluding remarks 321 14.9 Auxiliary results, lemmas, and proofs 322 14.9.1 Auxiliary Results 322 14.9.2 Lemmas 324 14.9.3 Proofs 325 14.10 Bibliographical notes 328 Appendix A: Convolution and kernel functions 331 A.1 Introduction 331 A.2 Convergence 332 A.2.1 Pointwise convergence 332 A.2.2 Convergence rate 337 A.2.3 Integrated error 339 A.3 Applications to probability 340 A.4 Lemmas 341 Appendix B Orthogonal functions 343 B.1 Introduction 343 B.2 Fourier series 345 B.3 Legendre series 352 B.4 Laguerre series 357 B.5 Hermite series 363 B.6 Wavelets 367 Appendix C: Probability and statistics 371 C.1 White noise 371 C.1.1 Discrete time 371 C.1.2 Continuous time 372 C.2 Convergence of random variables 373 C.3 Stochastic approximation 376 C.4 Order statistics 377 C.4.1 Distributions and moments 377 C.4.2 Spacings 379 C.4.3 Integration and random spacings 382 References 383 Index 399 Half-title......Page 3 Title......Page 5 Copyright......Page 6 Contents......Page 7 Dedication......Page 10 Preface......Page 11 1 Introduction......Page 13 2.1 The system......Page 15 2.2.1 The problem and the motivation for algorithms......Page 16 2.2.2 Simulation example......Page 19 2.3 Dynamic subsystem identification......Page 20 2.4 Bibliographic notes......Page 21 3.1 Motivation......Page 23 3.2 Consistency......Page 25 3.3 Applicable kernels......Page 26 3.4 Convergence rate......Page 28 3.6 Simulation example......Page 33 3.7.1 Lemmas......Page 36 3.7.2 Proofs......Page 37 3.8 Bibliographic notes......Page 41 4.1 Introduction......Page 42 4.2 Consistency and convergence rate......Page 43 4.3 Simulation example......Page 46 4.4.1 Lemmas......Page 47 4.4.2 Proofs......Page 49 4.5 Bibliographic notes......Page 55 5.2 Relation to stochastic approximation......Page 56 5.3 Consistency and convergence rate......Page 58 5.4 Simulation example......Page 61 5.5.1 Auxiliary results......Page 63 5.5.2 Lemmas......Page 65 5.6 Bibliographic notes......Page 70 6.1 Introduction......Page 71 6.2 Fourier series estimate......Page 73 6.3 Legendre series estimate......Page 76 6.4 Laguerre series estimate......Page 78 6.5 Hermite series estimate......Page 80 6.6 Wavelet estimate......Page 81 6.7 Local and global errors......Page 82 6.8 Simulation example......Page 83 6.9.1 Lemmas......Page 84 6.9.2 Proofs......Page 85 6.10 Bibliographic notes......Page 90 7.1 Introduction......Page 92 7.2.1 Motivation and estimates......Page 93 7.2.2 Consistency and convergence rate......Page 94 7.2.3 Simulation example......Page 96 7.3.1 Motivation......Page 97 7.3.2 Fourier series estimate......Page 98 7.3.3 Legendre series estimate......Page 100 7.4.1 Lemmas......Page 101 7.4.2 Proofs......Page 104 7.5 Bibliographic notes......Page 111 8.1 Identification problem......Page 113 8.1.2 Dynamic subsystem identification......Page 114 8.2 Kernel algorithm......Page 115 8.3 Orthogonal series algorithms......Page 118 8.4.1 Lemmas......Page 120 8.4.2 Proofs......Page 121 8.5 Bibliographic notes......Page 124 9.1 The system......Page 125 9.2.1 The problem and the motivation for algorithms......Page 126 9.2.2 Possible generalizations......Page 128 9.2.3 Monotonicity-preserving algorithms......Page 129 9.3.1 The motivation......Page 131 9.3.2 The algorithm......Page 132 9.4 Lemmas......Page 133 9.5 Bibliographic notes......Page 134 10.1.1 Introduction......Page 135 10.1.2 Differentiable characteristic......Page 136 10.1.3 Lipschitz characteristic......Page 137 10.2.2 Fourier series algorithm......Page 138 10.2.4 Hermite series algorithm......Page 140 10.3 Simulation example......Page 141 10.4.1 Lemmas......Page 142 10.4.2 Proofs......Page 146 10.5 Bibliographic notes......Page 154 11.1 Identification problem......Page 155 11.2 Nonlinear subsystem......Page 156 11.4 Lemmas......Page 158 11.5 Bibliographic notes......Page 160 12.1.1 Parallel nonlinear system......Page 161 12.1.2 Series-parallel models with nuisance characteristics......Page 169 12.1.3 Parallel-series models......Page 172 12.1.4 Generalized nonlinear block-oriented models......Page 174 12.2 Block-oriented systems with nonlinear dynamics......Page 185 12.2.1 Nonlinear models......Page 187 12.2.2 Identification algorithms: Nonlinear system identification......Page 191 12.2.3 Identification algorithms: Linear system identification......Page 198 12.2.4 Identification algorithms: the Gaussian input signal......Page 203 12.2.5 Sandwich systems with a Gaussian input signal......Page 217 12.2.6 Convergence of identification algorithms......Page 221 12.3 Concluding remarks......Page 230 12.4 Bibliographical notes......Page 232 13.1 Multivariate nonparametric regression......Page 234 13.2.1 Approximation by additive functions......Page 240 13.2.2 Additive regression......Page 245 13.3 Multivariate systems......Page 254 13.5 Bibliographic notes......Page 260 14.1 Introduction......Page 262 14.2.1 Semiparametric Hammerstein models......Page 264 14.2.2 Semiparametric Wiener models......Page 266 14.3 Statistical inference for semiparametric models......Page 267 14.3.1 Consistency of semiparametric estimates......Page 271 14.4 Statistical inference for semiparametric Wiener models......Page 276 14.4.1 Identification algorithms......Page 279 14.4.2 Convergence analysis......Page 284 14.4.3 Simulation examples......Page 290 14.4.4 Extensions......Page 294 14.5 Statistical inference for semiparametric Hammerstein models......Page 298 14.6 Statistical inference for semiparametric parallel models......Page 299 14.7.1 Average derivative estimation......Page 302 14.7.2 The average derivative estimate for the semiparametric Wiener model......Page 306 14.7.3 The average derivative estimate for the additive Wiener model......Page 312 14.7.4 The average derivative estimate for semiparametric multivariate Hammerstein models......Page 316 14.7.5 The average derivative estimate for semiparametric multivariate parallel models......Page 319 14.8 Concluding remarks......Page 321 14.9.1 Auxiliary Results......Page 322 14.9.2 Lemmas......Page 324 14.9.3 Proofs......Page 325 14.10 Bibliographical notes......Page 328 A.1 Introduction......Page 331 A.2.1 Pointwise convergence......Page 332 A.2.2 Convergence rate......Page 337 A.2.3 Integrated error......Page 339 A.3 Applications to probability......Page 340 A.4 Lemmas......Page 341 B.1 Introduction......Page 343 B.2 Fourier series......Page 345 B.3 Legendre series......Page 352 B.4 Laguerre series......Page 357 B.5 Hermite series......Page 363 B.6 Wavelets......Page 367 C.1.1 Discrete time......Page 371 C.1.2 Continuous time......Page 372 C.2 Convergence of random variables......Page 373 C.3 Stochastic approximation......Page 376 C.4.1 Distributions and moments......Page 377 C.4.2 Spacings......Page 379 C.4.3 Integration and random spacings......Page 382 References......Page 383 Index......Page 399 "Presenting a thorough overview of the theoretical foundations of nonparametric systems identification for nonlinear block-oriented systems, Wlodzimierz Greblicki and Miroslaw Pawlak show that nonparametric regression can be successfully applied to system identification, and they highlight what you can achieve in doing so." This book is aimed at researchers and practitioners in systems theory, signal processing, and communications. It will also appeal to researchers in fields such as mechanics, economics, and biology, where experimental data are used to obtain models of systems.