Adaptive Dynamic Programming for Control: Algorithms and Stability (Communications and Control Engineering)

There are many methods of stable controller design for nonlinear systems. In seeking to go beyond the minimum requirement of stability, Adaptive Dynamic Programming in Discrete Time approaches the challenging topic of optimal control for nonlinear systems using the tools of adaptive dynamic programming (ADP). The range of systems treated is extensive; affine, switched, singularly perturbed and time-delay nonlinear systems are discussed as are the uses of neural networks and techniques of value and policy iteration. The text features three main aspects of ADP in which the methods proposed for stabilization and for tracking and games benefit from the incorporation of optimal control methods: • infinite-horizon control for which the difficulty of solving partial differential Hamilton–Jacobi–Bellman equations directly is overcome, and proof provided that the iterative value function updating sequence converges to the infimum of all the value functions obtained by admissible control law sequences; • finite-horizon control, implemented in discrete-time nonlinear systems showing the reader how to obtain suboptimal control solutions within a fixed number of control steps and with results more easily applied in real systems than those usually gained from infinite-horizon control; • nonlinear games for which a pair of mixed optimal policies are derived for solving games both when the saddle point does not exist, and, when it does, avoiding the existence conditions of the saddle point. Non-zero-sum games are studied in the context of a single network scheme in which policies are obtained guaranteeing system stability and minimizing the individual performance function yielding a Nash equilibrium. In order to make the coverage suitable for the student as well as for the expert reader, Adaptive Dynamic Programming in Discrete Time: • establishes the fundamental theory involved clearly with each chapter devoted to a clearly identifiable control paradigm; • demonstrates convergence proofs of the ADP algorithms to deepen understanding of the derivation of stability and convergence with the iterative computational methods used; and • shows how ADP methods can be put to use both in simulation and in real applications. This text will be of considerable interest to researchers interested in optimal control and its applications in operations research, applied mathematics computational intelligence and engineering. Graduate students working in control and operations research will also find the ideas presented here to be a source of powerful methods for furthering their study. Adaptive Dynamic Programming for Control......Page 2 Background of This Book......Page 4 The Content of This Book......Page 5 Acknowledgments......Page 8 Contents......Page 10 1.1 Challenges of Dynamic Programming......Page 15 1.2 Background and Development of Adaptive Dynamic Programming......Page 17 1.2.1.1 Heuristic Dynamic Programming (HDP)......Page 18 1.2.1.2 Dual Heuristic Programming (DHP)......Page 19 1.2.2.1 Development of ADP Structures......Page 20 1.2.2.2 Development of Algorithms and Convergence Analysis......Page 23 1.2.2.3 Applications of ADP Algorithms......Page 24 1.3 Feedback Control Based on Adaptive Dynamic Programming......Page 25 1.4 Non-linear Games Based on Adaptive Dynamic Programming......Page 31 References......Page 33 2.2 Infinite-Horizon Optimal State Feedback Control Based on DHP......Page 40 2.2.1 Problem Formulation......Page 41 2.2.2 Infinite-Horizon Optimal State Feedback Control via DHP......Page 43 2.2.3 Simulations......Page 57 2.3.1 Problem Formulation......Page 65 2.3.2.1 NN Identification of the Unknown Nonlinear System......Page 67 2.3.2.2 Derivation of the Iterative ADP Algorithm......Page 70 2.3.2.3 Convergence Analysis of the Iterative ADP Algorithm......Page 71 2.3.2.4 NN Implementation of the Iterative ADP Algorithm Using GDHP Technique......Page 77 2.3.3 Simulations......Page 80 2.4.1 Problem Formulation......Page 84 2.4.2 Constrained Optimal Control Based on GHJB Equation......Page 86 2.4.3 Simulations......Page 91 2.5 Finite-Horizon Optimal State Feedback Control Based on HDP......Page 93 2.5.1 Problem Formulation......Page 95 2.5.2.1 Derivation and Properties of the Iterative ADP Algorithm......Page 97 2.5.2.2 The epsilon-Optimal Control Algorithm......Page 104 2.5.3 Simulations......Page 115 References......Page 119 3.2 Infinite-Horizon Optimal Tracking Control Based on HDP......Page 121 3.2.1 Problem Formulation......Page 122 3.2.2.1 System Transformation......Page 123 3.2.2.2 Derivation of the Iterative HDP Algorithm......Page 124 3.2.2.3 Summary of the Algorithm......Page 129 3.2.3 Simulations......Page 130 3.3 Infinite-Horizon Optimal Tracking Control Based on GDHP......Page 132 3.3.1 Problem Formulation......Page 135 3.3.2.1 Design and Implementation of Feedforward Controller......Page 138 3.3.2.2 Design and Implementation of Optimal Feedback Controller......Page 139 3.3.2.3 Convergence Characteristics of the Neural-Network Approximation Process......Page 147 3.3.3 Simulations......Page 149 3.4 Finite-Horizon Optimal Tracking Control Based on ADP......Page 150 3.4.1 Problem Formulation......Page 153 3.4.2.1 Derivation of the Iterative ADP Algorithm......Page 156 3.4.2.2 Convergence Analysis of the Iterative ADP Algorithm......Page 158 3.4.2.3 The epsilon-Optimal Control Algorithm......Page 162 3.4.2.5 Neural-Network Implementation of the Iterative ADP Algorithm via HDP Technique......Page 163 3.4.3 Simulations......Page 166 3.5 Summary......Page 170 References......Page 171 4.1 Introduction......Page 173 4.2.1 Problem Formulation......Page 174 4.2.2 Optimal State Feedback Control Using Delay Matrix......Page 175 4.2.2.1 Model Network......Page 184 4.2.2.3 Critic Network......Page 185 4.2.2.4 Action Network......Page 186 4.2.3 Simulations......Page 187 4.3.1 Problem Formulation......Page 189 4.3.2 Optimal Control Based on Iterative HDP......Page 192 4.3.3 Simulations......Page 198 4.4.1 Problem Formulation......Page 200 4.4.2 Optimal Control Based on Improved Iterative ADP......Page 202 4.4.3 Simulations......Page 208 4.5 Summary......Page 209 References......Page 210 5.2 Problem Formulation......Page 212 5.3 Optimal Tracking Control Based on Improved Iterative ADP Algorithm......Page 213 5.4 Simulations......Page 224 References......Page 231 6.2 Optimal Robust Feedback Control for Unknown General Nonlinear Systems......Page 233 6.2.2 Data-Based Robust Approximate Optimal Tracking Control......Page 234 6.2.3 Simulations......Page 246 6.3.1 Problem Formulation......Page 252 6.3.2 Robust Approximate Optimal Control Based on ADP Algorithm......Page 253 6.3.3 Simulations......Page 260 6.4 Summary......Page 263 References......Page 264 7.1 Introduction......Page 266 7.2.1 Problem Description......Page 267 7.2.2 Optimal Feedback Control Based on Two-Stage ADP Algorithm......Page 268 7.2.3 Simulations......Page 277 7.3.1 Problem Formulation......Page 280 7.3.2 Optimal Controller Design for a Class of Descriptor Systems......Page 282 7.3.3 Simulations......Page 288 7.4.1 Problem Formulation......Page 290 7.4.2.1 Algorithm Design......Page 292 7.4.2.2 Neural Network Approximation......Page 295 7.5.1 Problem Formulation......Page 297 7.5.2 Optimal Controller Design for Constrained Systems via SNAC......Page 301 7.5.3 Simulations......Page 308 References......Page 315 8.2 Zero-Sum Differential Games for a Class of Discrete-Time 2-D Systems......Page 317 8.2.1 Problem Formulation......Page 318 8.2.2 Data-Based Optimal Control via Iterative ADP Algorithm......Page 325 8.2.2.1 The Derivation of Data-Based Iterative ADP Algorithm......Page 326 8.2.2.2 Properties of Data-Based Iterative ADP Algorithm......Page 327 8.2.2.4 Critic Network......Page 334 8.2.2.5 Action Networks......Page 335 8.2.3 Simulations......Page 336 8.3 Zero-Sum Games for a Class of Discrete-Time Systems via Model-Free ADP......Page 339 8.3.1 Problem Formulation......Page 340 8.3.2 Data-Based Optimal Output Feedback Control via ADP Algorithm......Page 342 8.3.3 Simulations......Page 349 References......Page 351 9.1 Introduction......Page 353 9.2.1 Problem Formulation......Page 354 9.2.2.1 Derivation of the Iterative ADP Method......Page 355 9.2.2.2 The Iterative ADP Algorithm......Page 357 9.2.2.3 Properties of the Iterative ADP Algorithm......Page 358 9.2.3 Simulations......Page 363 9.3 Finite Horizon Zero-Sum Games for a Class of Nonlinear Systems......Page 366 9.3.1 Problem Formulation......Page 368 9.3.2 Finite Horizon Optimal Control of Nonaffine Nonlinear Zero-Sum Games......Page 370 9.3.3 Simulations......Page 378 9.4 Non-Zero-Sum Games for a Class of Nonlinear Systems Based on ADP......Page 380 9.4.1 Problem Formulation of Non-Zero-Sum Games......Page 381 9.4.2 Optimal Control of Nonlinear Non-Zero-Sum Games Based on ADP......Page 384 9.4.3 Simulations......Page 395 9.5 Summary......Page 399 References......Page 400 10.1 Introduction......Page 402 10.2.1 Problem Formulation......Page 403 10.2.2.1 Adaptive Critic Designs for Problems with Finite Action Space......Page 405 10.2.2.2 Self-learning Call Admission Control for CDMA Cellular Networks......Page 409 10.2.3 Simulations......Page 413 10.3.1 Problem Formulation......Page 419 10.3.2 Self-learning Neural Network Control for Both Engine Torque and Exhaust Air-Fuel Ratio......Page 420 10.3.3.1 Critic Network......Page 422 10.3.3.3 Simulation Results......Page 424 10.4 Summary......Page 426 References......Page 427 Index......Page 430 There are many methods of stable controller design for nonlinear systems. In seeking to go beyond the minimum requirement of stability, Adaptive Dynamic Programming for Control approaches the challenging topic of optimal control for nonlinear systems using the tools of adaptive dynamic programming (ADP). The range of systems treated is extensive; affine, switched, singularly perturbed and time-delay nonlinear systems are discussed as are the uses of neural networks and techniques of value and policy iteration.^ The text features three main aspects of ADP in which the methods proposed for stabilization and for tracking and games benefit from the incorporation of optimal control methods: • infinite-horizon control for which the difficulty of solving partial differential Hamilton–Jacobi–Bellman equations directly is overcome, and proof provided that the iterative value function updating sequence converges to the infimum of all the value functions obtained by admissible control law sequences; • finite-horizon control, implemented in discrete-time nonlinear systems showing the reader how to obtain suboptimal control solutions within a fixed number of control steps and with results more easily applied in real systems than those usually gained from infinte-horizon control; • nonlinear games for which a pair of mixed optimal policies are derived for solving games both when the saddle point does not exist, and, when it does,^ avoiding the existence conditions of the saddle point. Non-zero-sum games are studied in the context of a single network scheme in which policies are obtained guaranteeing system stability and minimizing the individual performance function yielding a Nash equilibrium. In order to make the coverage suitable for the student as well as for the expert reader, Adaptive Dynamic Programming for Control : • establishes the fundamental theory involved clearly with each chapter devoted to a clearly identifiable control paradigm; • demonstrates convergence proofs of the ADP algorithms to deepen undertstanding of the derivation of stability and convergence with the iterative computational methods used; and • shows how ADP methods can be put to use both in simulation and in real applications.^ This text will be of considerable interest to researchers interested in optimal control and its applications in operations research, applied mathematics computational intelligence and engineering. Graduate students working in control and operations research will also find the ideas presented here to be a source of powerful methods for furthering their study. The Communications and Control Engineering series reports major technological advances which have potential for great impact in the fields of communication and control. It reflects research in industrial and academic institutions around the world so that the readership can exploit new possibilities as they become available. There are many methods of stable controller design for nonlinear systems. In seeking to go beyond the minimum requirement of stability, Adaptive Dynamic Programming for Control approaches the challenging topic of optimal control for nonlinear systems using the tools of adaptive dynamic programming (ADP). The range of systems treated is extensive; affine, switched, singularly perturbed and time-delay nonlinear systems are discussed as are the uses of neural networks and techniques of value and policy iteration. The text features three main aspects of ADP in which the methods proposed for stabilization and for tracking and games benefit from the incorporation of optimal control methods: • infinite-horizon control for which the difficulty of solving partial differential Hamilton-Jacobi-Bellman equations directly is overcome, and proof provided that the iterative value function updating sequence converges to the infimum of all the value functions obtained by admissible control law sequences; • finite-horizon control, implemented in discrete-time nonlinear systems showing the reader how to obtain suboptimal control solutions within a fixed number of control steps and with results more easily applied in real systems than those usually gained from infinte-horizon control; • nonlinear games for which a pair of mixed optimal policies are derived for solving games both when the saddle point does not exist, and, when it does, avoiding the existence conditions of the saddle point. Non-zero-sum games are studied in the context of a single network scheme in which policies are obtained guaranteeing system stability and minimizing the individual performance function yielding a Nash equilibrium. In order to make the coverage suitable for the student as well as for the expert reader, Adaptive Dynamic Programming for Control: • establishes the fundamental theory involved clearly with each chapter devoted to a clearly identifiable control paradigm; • demonstrates co nvergence proofs of the ADP algorithms to deepen undertstanding of the derivation of stability and convergence with the iterative computational methods used; and • shows how ADP methods can be put to use both in simulation and in real applications. This text will be of considerable interest to researchers interested in optimal control and its applications in operations research, applied mathematics computational intelligence and engineering. Graduate students working in control and operations research will also find the ideas presented here to be a source of powerful methods for furthering their study. The Communications and Control Engineering series reports major technological advances which have potential for great impact in the fields of communication and control. It reflects research in industrial and academic institutions around the world so that the readership can exploit new possibilities as they become available Optimal Stabilization Control for Discrete-time Systems Optimal Tracking Control for Discrete-time Systems Optimal Stabilization Control for Nonlinear Systems with Time Delays Optimal Tracking Control for Nonlinear Systems with Time-delays Optimal Feedback Control for Continuous-time Systems via ADP Several Special Optimal Feedback Control Designs Based on ADP Zero-sum Games for Discrete-time Systems Based on Model-free ADP Nonlinear Games for a Class of Continuous-time Systems Based on ADP Other Applications of ADP.