Reinforcement Learning. Theory and Applications

Издательство InTech, 2011, -434 pp. Brains rule the world, and brain-like computation is increasingly used in computers and electronic devices. Brain-like computation is about processing and interpreting data or directly putting forward and performing actions. Learning is a very important aspect. This book is on reinforcement learning which involves performing actions to achieve a goal. Two other learning paradigms exist. Supervised learning has initially been successful in prediction and classification tasks, but is not brain-like. Unsupervised learning is about understanding the world by passively mapping or clustering given data according to some order principles, and is associated with the cortex in the brain. In reinforcement learning an agent learns by trial and error to perform an action to receive a reward, thereby yielding a powerful method to develop goal-directed action strategies. It is predominately associated with the basal ganglia in the brain. The first 11 chapters of this book, Theory, describe and extend the scope of reinforcement learning. The remaining 11 chapters, Applications, show that there is already wide usage in numerous fields. Reinforcement learning can tackle control tasks that are too complex for traditional, hand-designed, non-learning controllers. As learning computers can deal with technical complexities, the tasks of human operators remain to specify goals on increasingly higher levels. This book shows that reinforcement learning is a very dynamic area in terms of theory and applications and it shall stimulate and encourage new research in this field. We would like to thank all contributors to this book for their research and effort. Summary of Theory: Chapters 1 and 2 create a link to supervised and unsupervised learning, respectively, by regarding reinforcement learning as a prediction problem, and chapter 3 looks at fuzzycontrol with a reinforcement-based genetic algorithm. Reinforcement algorithms are modified in chapter 4 for future parallel and quantum computing, and in chapter 5 for a more general class of state-action spaces, described by grammars. Then follow biological views; in chapter 6 how reinforcement learning occurs on a single neuron level by considering the interaction between a spatio-temporal learning rule and Hebbian learning, and in a global brain view of chapter 7, unsupervised learning is depicted as a means of data pre-processing and arrangement for reinforcement algorithms. A table presents a ready-to-implement description of standard reinforcement learning algorithms. The following chapters consider multi agent systems where a single agent has only partial view of the entire system. Multiple agents can work cooperatively on a common goal, as considered in chapter 8, or rewards can be individual but interdependent, such as in game play, as considered in chapters 9, 10 and 11. Summary of Applications: Chapter 12 continues with game applications where a robot cup middle size league robot learns a strategic soccer move. A dialogue manager for man-machine dialogues in chapter 13 interacts with humans by communication and database queries, dependent on interaction strategies that govern the Markov decision processes. Chapters 14, 15, 16 and 17 tackle control problems that may be typical for classical methods of control like PID controllers and hand-set rules. However, traditional methods fail if the systems are too complex, timevarying, if knowledge of the state is imprecise, or if there are multiple objectives. These chapters report examples of computer applications that are tackled only with reinforcement learning such as water allocation improvement, building environmental control, chemical processing and industrial process control. The reinforcement-controlled systems may continue learning during operation. The next three chapters involve path optimization. In chapter 18, internet routers explore different links to find more optimal routes to a destination address. Chapter 19 deals with optimizing a travel sequence w.r.t. both time and distance. Chapter 20 proposes an untypical application of path optimization: a path from a given pattern to a target pattern provides a distance measure. An unclassified medical image can thereby be classified dependent on whether a path from it is shorter to an image of healthy or unhealthy tissue, specifically considering lung nodules classification using 3D geometric measures extracted from the lung lesions Computerized Tomography (CT) images. Chapter 21 presents a physicians' decision support system for diagnosis and treatment, involving a knowledgebase server. In chapter 22 a reinforcement learning sub-module improves the efficiency for the exchange of messages in a decision support system in air traffic management. Neural Forecasting Systems Reinforcement learning in system identification Reinforcement Evolutionary Learning for Neuro-Fuzzy Controller Design Superposition-Inspired Reinforcement Learning and Quantum Reinforcement Learning An Extension of Finite-state Markov Decision Process and an Application of Grammatical Inference Interaction between the Spatio-Temporal Learning Rule (non Hebbian) and Hebbian in Single Cells: A cellular mechanism of reinforcement learning Reinforcement Learning Embedded in Brains and Robots Decentralized Reinforcement Learning for the Online Optimization of Distributed System Multi-Automata Learning Abstraction for Genetics-based Reinforcement Learning Dynamics of the Bush-Mosteller learning algorithm in 2x2 games Modular Learning Systems for Behavior Acquisition in Multi-Agent Environment Optimising Spoken Dialogue Strategies within the Reinforcement Learning Paradigm Water Allocation Improvement in River Basin Using Adaptive Neural Fuzzy Reinforcement Learning Approach Reinforcement Learning for Building Environmental Control Model-Free Learning Control of Chemical Processes Reinforcement Learning-Based Supervisory Control Strategy for a Rotary Kiln Process Inductive Approaches based on Trial/Error Paradigm for Communications Network The Allocation of Time and Location Information to Activity-Travel Sequence Data by means of Reinforcement Learning Application on Reinforcement Learning for Diagnosis based on Medical Image RL based Decision Support System for u-Healthcare Environment Reinforcement Learning to Support Meta-Level Control in Air Traffic Management Abstraction may appear a trivial task for humans and the positive results from this work intuitive, but abstraction has not been routinely used in genetics-based reinforcement learning. One reason is that the time each iteration requires is an important consideration and abstraction increases the time for each iteration. Typically XCS takes 20 minutes to play 1000 games (and remains constant), mXCS with abstraction takes 20 minutes for 100 games (although this can vary greatly depending on the choice of parameters) and the Q-Learning algorithm ranges from 5 minutes for 1000 games initially to 90 minutes for 1000 games after 100,000 games training. However, given a fixed amount of time to train all three algorithms mXCS with abstraction would perform the best, once the initial base rules were found. The Q-Learning algorithm has to visit every single state at least once in order to form a successful playing strategy. Whilst the Q-Learning system would ultimately play a very good game, weeks of computation failed to achieve the level of success the Abstraction algorithm had in a very short space of time (hours rather than weeks). Although better Q-learning algorithms (including generalization capabilities) exist (Sutton & Barto, 1998) this choice of benchmark algorithm showed the scale of the problem, which is difficult to calculate. The improvement in abstraction performance from standard XCS to the modified XCS was due to using simpler reinforcement learning. The Widrow-Hoff delta rule converges much faster, which for simpler domains that can be solved easily is beneficial. However, slower and more graceful learning may be required in complex domains when interacting with higher level features. The abstracted rules allow the system to play on states as a whole, including those that have not been encountered, where these states contain a known pattern. This is useful in data-mining, but with the inherent dangers of interpolation and extrapolation. The abstracted rule-base is also compact as an abstracted rule covers more states than either a generalized LCS rule or a Qlearning state. Unique states may still be covered by the base rules. Abstraction has been shown to give an improvement in a complex, but structured domain. It is anticipated that the Abstraction algorithm would be suited to other domains containing repeated patterns The research leading to this paper was conducted at the Shiraz University, Iran. The measured data were collected by Fars Regional Water Authority? Iran, and Fars Agricultural Research Center? Iran. The authors are grateful to Dr. B. Zahraie from University of Tehran, Sh. Araghi? Nejhad and Reza Karachian Ph. D. Candidates of Amir Kabir University. Water and Environment Research and Development (WE-R & D) office is also greatly appreciated