Adaptive Filter Theory

Adaptive Filter Theory (3rd Ed.)......Page 1 Contents......Page 2 1 The Filtering Problem......Page 9 2 Adpative Filters......Page 10 3 Linear Filter Structures......Page 12 4 Approaches to Development of Linear Adaptive Filtering Algorithms......Page 17 5 Real & Complex Forms of Adaptive Filters......Page 22 6 Nonlinear Adaptive Filters......Page 23 7 Applications......Page 26 8 Some Historical Notes......Page 75 Part 1 Background Material......Page 86 1.1 z-Transform......Page 87 1.2 Linear Time-Invariant Filters......Page 89 1.3 Minimum Phase Filters......Page 94 1.5 Implementing Convolutions using DFT......Page 95 1.6 Discrete Cosine Transform......Page 101 1.7 Summary & Discussion......Page 102 Problems......Page 103 Ch2 Stationary Processes & Models......Page 104 2.1 Partial Characterization of Discrete-Time Stochastic Process......Page 105 2.2 Mean Ergodic Theorem......Page 106 2.3 Correlation Matrix......Page 108 2.4 Correlation Matrix of Sine Wave Plus Noise......Page 114 2.5 Stochastic Models......Page 116 2.6 Wold Decomposition......Page 123 2.7 Asymptotic Stationary of Autoregressive Process......Page 124 2.8 Yule-Walker Equations......Page 126 2.9 Computer Experiment: Autoregressive Process of Order 2......Page 128 2.10 Selecting the Model Order......Page 136 2.11 Complex Gaussian Processes......Page 138 2.12 Summary & Discussion......Page 140 Problems......Page 141 3.1 Power Spectral Density......Page 144 3.2 Properties of Power Spectral Density......Page 146 3.3 Transmission of Stationary Process through Linear Filter......Page 148 3.4 Cramer Spectral Representation for Stationary Process......Page 152 3.5 Power Spectrum Estimation......Page 154 3.6 Other Statistical Characteristics of Stationary Process......Page 157 3.7 Polyspectra......Page 158 3.8 Spectral-Correlation Density......Page 162 3.9 Summary & Discussion......Page 165 Problems......Page 166 4.1 The Eigenvalue Problem......Page 168 4.2 Properties of Eigenvalues & Eigenvectors......Page 170 4.3 Low-Rank Modeling......Page 184 4.4 Eigenfilters......Page 189 4.5 Eigenvalue Computations......Page 192 4.6 Summary & Discussion......Page 195 Problems......Page 196 Part 2 Linear Optimum Filtering......Page 201 5.1 Linear Optimum Filtering: Problem Statement......Page 202 5.2 Principle of Orthogonality......Page 205 5.3 Minimum Mean-Squared Error......Page 209 5.4 Wiener-Hoff Equations......Page 211 5.5 Error-Performance Surface......Page 214 5.6 Numerical Example......Page 218 5.7 Channel Equalization......Page 225 5.8 Linearly Constrained Minimum Variance Filter......Page 228 5.9 Generalized Sidelobe Cancelers......Page 235 5.10 Summary & Discussion......Page 243 Problems......Page 244 Ch6 Linear Prediction......Page 249 6.1 Forward Linear Prediction......Page 250 6.2 Backward Linear Prediction......Page 256 6.3 Levinson-Durbin Algorithm......Page 262 6.4 Properties of Prediction-Error Filters......Page 270 6.5 Schur-Cohn Test......Page 279 6.6 Autoregressive Modeling of Stationary Stochastic Process......Page 281 6.7 Cholesky Factorization......Page 284 6.8 Lattice Predictors......Page 288 6.9 Joint-Process Estimation......Page 294 6.10 Block Estimation......Page 298 6.11 Summary & Discussion......Page 301 Problems......Page 303 Ch7 Kalman Filters......Page 310 7.1 Recursive Minimum Mean-Square Estimation for Scalar Random Variables......Page 311 7.2 Statement of Kalman Filtering Problem......Page 314 7.3 The Innovations Process......Page 315 7.4 Estimation of the State using the Innovations Process......Page 318 7.5 Filtering......Page 325 7.7 Summary of Kalman Filter......Page 328 7.8 Variants of Kalman Filter......Page 330 7.9 Extended Kalman Filter......Page 336 7.10 Summary & Discussion......Page 341 Problems......Page 342 Part 3 Linear Adaptive Filtering......Page 346 8.1 Some Preliminaries......Page 347 8.2 Steepest-Descent Algorithm......Page 349 8.3 Stability of Steepest-Descent Algorithm......Page 351 8.4 Example......Page 358 Problems......Page 370 9.1 Overview of Structure & Operation of Least-Mean-Square Algorithm......Page 373 9.2 Least-Mean-Square Adaptation Algorithm......Page 375 9.3 Examples......Page 380 9.4 Stability & Performance Analysis of LMS Algorithm......Page 398 9.5 Summary of LMS Algorithm......Page 413 9.6 Computer Experiment on Adaptive Prediction......Page 414 9.7 Computer Experiment on Adaptive Equalization......Page 420 9.8 Computer Experiment on Minimum-Variance Distortionless Response Beamformer......Page 429 9.9 Directionality of Convergence of LMS Algorithm for Non-White Inputs......Page 433 9.10 Robustness of LMS Algorithm......Page 435 9.11 Normalized LMS Algorithm......Page 440 9.12 Summary & Discussion......Page 446 Problems......Page 447 Ch10 Frequency-Domain Adaptive Filters......Page 453 10.1 Block Adaptive Filters......Page 454 10.2 Fast LMS Algorithm......Page 459 10.3 Unconstrained Frequency-Domain Adaptive Filtering......Page 465 10.4 Self-Orthogonalizing Adaptive Filters......Page 466 10.5 Computer Experiment on Adaptive Equalization......Page 477 10.6 Classification of Adaptive Filtering Algorithms......Page 485 10.7 Summary & Discussion......Page 486 Problems......Page 487 11.1 Statement of Linear Least-Square Estimation Problem......Page 491 11.2 Data Windowing......Page 494 11.3 Principle of Orthogonality (Revisited)......Page 495 11.4 Minimum Sum of Error Squares......Page 499 11.5 Normal Equations & Linear Least-Squares Filters......Page 500 11.6 Time-Averaged Correlation Matrix......Page 503 11.7 Reformulation of Normal Equations in Terms of Data Matrices......Page 505 11.8 Properties of Least-Squares Estimates......Page 510 11.9 Parametric Spectrum Estimation......Page 514 11.10 Singular Value Decomposition......Page 524 11.11 Pseudoinverse......Page 532 11.12 Interpretation of Singular Values & Singular Vectors......Page 533 11.13 Minimum Norm Solution to Linear Least-Square Problem......Page 534 11.14 Normalized LMS Algorithm Viewed as Minimum-Norm Solution to Underdetermined Least-Squares Estimation Problem......Page 538 11.15 Summary & Discussion......Page 540 Problems......Page 541 Ch12 Rotations and Reflections......Page 544 12.1 Plane Rotations......Page 545 12.2 Two-Sided Jacobi Algorithm......Page 546 12.3 Cyclic Jacobi Algorithm......Page 552 12.4 Householder Transformation......Page 556 12.5 The QR Algorithm......Page 559 12.6 Summary & Discussion......Page 566 Problems......Page 568 Ch13 Recursive Least-Squares Algorithm......Page 570 13.1 Some Preliminaries......Page 571 13.2 Matrix Inversion Lemma......Page 573 13.3 Exponentially Weighted Recursive Least-Squares Algorithm......Page 574 13.4 Update Recursion for Sum of Weighted Error Squares......Page 579 13.5 Example: Single-Weight Adaptive Noise Canceler......Page 580 13.6 Convergence Analysis of RLS Algorithm......Page 581 13.7 Computer Experiment on Adaptive Equalization......Page 588 13.8 State-Space Formulation of RLS Problem......Page 591 Problems......Page 595 14.1 Square-Root Kalman Filters......Page 597 14.2 Building Square-Root Adaptive Filtering Algorithms on their Kalman Filter Counterparts......Page 605 14.3 QR-RLS Algorithm......Page 606 14.4 Extended QR-RLS Algorithm......Page 622 14.5 Adaptive Beamforming......Page 625 14.6 Inverse QR-RLS Algorithm......Page 632 14.7 Summary & Discussion......Page 635 Problems......Page 636 Ch15 Order-Recursive Adaptive Filters......Page 638 15.1 Adaptive Forward Linear Prediction......Page 639 15.2 Adaptive Backward Linear Prediction......Page 642 15.3 Conversion Factor......Page 644 15.4 Least-Squares Lattice Predictor......Page 648 15.5 Angle-Normalized Estimation Errors......Page 661 15.6 First-Order State-Space Models for Lattice Filtering......Page 663 15.7 QR-Decomposition-Based Least-Square Lattice Filters......Page 668 15.8 Fundamental Properties of QRD-LSL Filter......Page 675 15.9 Computer Experiment on Adaptive Equalization......Page 680 15.10 Extended QRD-LSL Algorithm......Page 685 15.11 Recursive Least-Squares Lattice Filters using A Posteriori Estimation Errors......Page 687 15.12 Recursive LSL Filters using A Priori Estimation Errors with Error Feedback......Page 691 15.13 Computation of Least-Squares Weight Vector......Page 694 15.14 Computer Experiment on Adaptive Prediction......Page 699 15.15 Other Variants of Least-Squares Lattice Filters......Page 701 15.16 Summary & Discussion......Page 702 Problems......Page 704 Ch16 Tracking of Time-Varying Systems......Page 709 16.1 Markov Model for System Identification......Page 710 16.2 Degree of Nonstationarity......Page 713 16.3 Criteria for Tracking Assessment......Page 714 16.4 Tracking Performance of LMS Algorithm......Page 716 16.5 Tracking Performance of RLS Algorithm......Page 719 16.6 Comparison of Tracking Performance of LMS & RLS Algorithms......Page 724 16.7 Adaptive Recovery of Chirped Sinusoidal in Noise......Page 727 16.8 How to Improve Tracking Behavior of RLS Algorithm......Page 734 16.9 Computer Experiment on System Identification......Page 737 16.10 Automatic Tuning of Adaptation Constants......Page 739 16.11 Summary & Discussion......Page 744 Problems......Page 745 Ch17 Fine-Precision Effects......Page 746 17.1 Quantization Errors......Page 747 17.2 Least-Mean-Square Algorithm......Page 749 17.3 Recursive Least-Squares Algorithm......Page 759 17.4 Square-Root Adaptive Filters......Page 765 17.5 Order-Recursive Adaptive Filters......Page 768 17.6 Fast Transversal Filters......Page 771 17.7 Summary & Discussion......Page 775 Problems......Page 777 Part 4 Nonlinear Adaptive Filtering......Page 779 Ch18 Blind Deconvolution......Page 780 18.1 Theoretical & Practical Considerations......Page 781 18.2 Bussgang Algorithm for Blind Equalization of Real Baseband Channels......Page 784 18.3 Extension of Bussgang Algorithms to Complex Baseband Channels......Page 799 18.4 Special Cases of Bussgang Algorithm......Page 800 18.5 Blind Channel Identification & Equalization using Polyspectra......Page 804 18.6 Advantage & Disadvantage of HOS-Based Deconvolution Algorithms......Page 810 18.7 Channel Identification using Cyclostationary Statistics......Page 811 18.8 Subspace Decomposition for Fractionally-Spaced Blind Identification......Page 812 18.9 Summary & Discussion......Page 821 Problems......Page 822 Ch19 Back-Propagation Learning......Page 825 19.1 Models of A Neuon......Page 826 19.2 Multilayer Perception......Page 830 19.3 Complex Back-Propagation Algorithm......Page 832 19.4 Back-Propagation Algorithm for Real Parameters......Page 845 19.5 Universal Approximation Theorem......Page 846 19.6 Network Complexity......Page 848 19.7 Filtering Applications......Page 850 19.8 Summary & Discussion......Page 860 Problems......Page 862 Ch20 Radial Basis Funuction Networks......Page 863 20.1 Structure of RBF Networks......Page 864 20.2 Radial-Basis Functions......Page 866 20.3 Fixed Centers Selected at Random......Page 867 20.4 Recursive Hybrid Learning Procedure......Page 870 20.5 Stochastic Gradient Approach......Page 871 20.6 Universal Approximation Theorem (Revisited)......Page 873 20.7 Filtering Applications......Page 874 20.8 Summary & Discussion......Page 879 Problems......Page 881 A.1 Cauthy-Riemann Equations......Page 883 A.2 Cauthy's Intergral Formula......Page 885 A.3 Laurent's Series......Page 887 A.4 Singularities & Residues......Page 889 A.5 Cauthy's Residue Theorem......Page 890 A.6 Principle of Argument......Page 892 A.7 Inversion Integral for z-Transform......Page 896 A.8 Parseval's Theorem......Page 897 B.1 Basic Definitions......Page 898 B.2 Examples......Page 900 B.3 Relation between Derivative with respect to Vector & Gradient Vector......Page 902 C.1 Optimization involving Single Equality Constraint......Page 903 C.2 Optimization involving Multiple Equality Constraints......Page 905 D.1 Likelihood Function......Page 907 D.3 Properties of Maximum-Likelihood Estimators......Page 909 D.4 Conditional Mean Estimator......Page 910 E.1 Maximum-Entropy Spectrum......Page 913 E.2 Computation of Mean Spectrum......Page 918 F.1 Fast MVDR Spectrum Computation......Page 920 F.2 Comparison of MVDR & MEM Spectra......Page 922 AppG Gradient Adaptive Lattice Algorithm......Page 923 AppH Solution of the Difference Equation (9.75)......Page 927 I.1 Iterative Solution for Weight-Error Vector......Page 929 I.2 Series Expansion of Weight-Error Correlation Matrix......Page 930 J.1 Definition......Page 932 J.2 Chi-Square Distribution as Special Case......Page 933 J.4 Expectation of Inverse Correlation Matrix......Page 935 Text Conventions......Page 936 Abbreviations......Page 940 Principal Symbols......Page 941 Bibliography......Page 949 Index......Page 986