Hands-on reinforcement learning for games : implementing self-learning agents in games using artificial intelligence techniques

BExplore reinforcement learning (RL) techniques to build cutting-edge games using Python libraries such as PyTorch, OpenAI Gym, and TensorFlow/b h4Key Features/h4 ulliGet to grips with the different reinforcement and DRL algorithms for game development /li liLearn how to implement components such as artificial agents, map and level generation, and audio generation /li liGain insights into cutting-edge RL research and understand how it is similar to artificial general research/li/ul h4Book Description/h4 With the increased presence of AI in the gaming industry, developers are challenged to create highly responsive and adaptive games by integrating artificial intelligence into their projects. This book is your guide to learning how various reinforcement learning techniques and algorithms play an important role in game development with Python. Starting with the basics, this book will help you build a strong foundation in reinforcement learning for game development. Each chapter will assist you in implementing different reinforcement learning techniques, such as Markov decision processes (MDPs), Q-learning, actor-critic methods, SARSA, and deterministic policy gradient algorithms, to build logical self-learning agents. Learning these techniques will enhance your game development skills and add a variety of features to improve your game agent's productivity. As you advance, you'll understand how deep reinforcement learning (DRL) techniques can be used to devise strategies to help agents learn from their actions and build engaging games. By the end of this book, you'll be ready to apply reinforcement learning techniques to build a variety of projects and contribute to open source applications. h4What you will learn/h4 ulliUnderstand how deep learning can be integrated into an RL agent /li liExplore basic to advanced algorithms commonly used in game development /li liBuild agents that can learn and solve problems in all types of environments /li liTrain a Deep Q-Network (DQN) agent to solve the CartPole balancing problem /li liDevelop game AI agents by understanding the mechanism behind complex AI /li liIntegrate all the concepts learned into new projects or gaming agents/li/ul h4Who this book is for/h4 If you're a game developer looking to implement AI techniques to build next-generation games from scratch, this book is for you. Machine learning and deep learning practitioners, and RL researchers who want to understand how to use self-learning agents in the game domain will also find this book useful. Knowledge of game development and Python programming experience are required Cover Title Page Copyright and Credits Dedication About Packt Contributors Table of Contents Preface Section 1: Exploring the Environment Chapter 1: Understanding Rewards-Based Learning Technical requirements Understanding rewards-based learning The elements of RL The history of RL Why RL in games? Introducing the Markov decision process The Markov property and MDP Building an MDP Using value learning with multi-armed bandits Coding a value learner Implementing a greedy policy Exploration versus exploitation Exploring Q-learning with contextual bandits Implementing a Q-learning agent Removing discounted rewards Summary Questions Chapter 2: Dynamic Programming and the Bellman Equation Introducing DP Regular programming versus DP Enter DP and memoization Understanding the Bellman equation Unraveling the finite MDP The Bellman optimality equation Building policy iteration Installing OpenAI Gym Testing Gym Policy evaluation Policy improvement Building value iteration Playing with policy versus value iteration Exercises Summary Chapter 3: Monte Carlo Methods Understanding model-based and model-free learning Introducing the Monte Carlo method Solving for Implementing Monte Carlo Plotting the guesses Adding RL Monte Carlo control Playing the FrozenLake game Using prediction and control Incremental means Exercises Summary Chapter 4: Temporal Difference Learning Understanding the TCA problem Introducing TDL Bootstrapping and backup diagrams Applying TD prediction TD(0) or one-step TD Tuning hyperparameters Applying TDL to Q-learning Exploring TD(0) in Q-learning Exploration versus exploitation revisited Teaching an agent to drive a taxi Running off- versus on-policy Exercises Summary Chapter 5: Exploring SARSA Exploring SARSA on-policy learning Using continuous spaces with SARSA Discretizing continuous state spaces Expected SARSA Extending continuous spaces Working with TD (λ) and eligibility traces Backward views and eligibility traces Understanding SARSA (λ) SARSA lambda and the Lunar Lander Exercises Summary Section 2: Exploiting the Knowledge Chapter 6: Going Deep with DQN DL for RL DL frameworks for DRL Using PyTorch for DL Computational graphs with tensors Training a neural network – computational graph Building neural networks with Torch Understanding DQN in PyTorch Refreshing the environment Partially observable Markov decision process Constructing DQN The replay buffer The DQN class Calculating loss and training Exercising DQN Revisiting the LunarLander and beyond Exercises Summary Chapter 7: Going Deeper with DDQN Understanding visual state Encoding visual state Introducing CNNs Working with a DQN on Atari Adding CNN layers Introducing DDQN Double DQN or the fixed Q targets Dueling DQN or the real DDQN Extending replay with prioritized experience replay Exercises Summary Chapter 8: Policy Gradient Methods Understanding policy gradient methods Policy gradient ascent Introducing REINFORCE Using advantage actor-critic Actor-critic Training advantage AC Building a deep deterministic policy gradient Training DDPG Exploring trust region policy optimization Conjugate gradients Trust region methods The TRPO step Exercises Summary Chapter 9: Optimizing for Continuous Control Understanding continuous control with Mujoco Introducing proximal policy optimization The hows of policy optimization PPO and clipped objectives Using PPO with recurrent networks Deciding on synchronous and asynchronous actors Using A2C Using A3C Building actor-critic with experience replay Exercises Summary Chapter 10: All about Rainbow DQN Rainbow – combining improvements in deep reinforcement learning Using TensorBoard Introducing distributional RL Back to TensorBoard Understanding noisy networks Noisy networks for exploration and importance sampling Unveiling Rainbow DQN When does training fail? Exercises Summary Chapter 11: Exploiting ML-Agents Installing ML-Agents Building a Unity environment Building for Gym wrappers Training a Unity environment with Rainbow Creating a new environment Coding an agent/environment Advancing RL with ML-Agents Curriculum learning Behavioral cloning Curiosity learning Training generalized reinforcement learning agents Exercises Summary Chapter 12: DRL Frameworks Choosing a framework Introducing Google Dopamine Playing with Keras-RL Exploring RL Lib Using TF-Agents Exercises Summary Section 3: Reward Yourself Chapter 13: 3D Worlds Reasoning on 3D worlds Training a visual agent Generalizing 3D vision ResNet for visual observation encoding Challenging the Unity Obstacle Tower Challenge Pre-training the agent Prierarchy – implicit hierarchies Exploring Habitat – embodied agents by FAIR Installing Habitat Training in Habitat Exercises Summary Chapter 14: From DRL to AGI Learning meta learning Learning 2 learn Model-agnostic meta learning Training a meta learner Introducing meta reinforcement learning MAML-RL Using hindsight experience replay Imagination and reasoning in RL Generating imagination Understanding imagination-augmented agents Exercises Summary Other Books You May Enjoy Index The AI revolution is here and it is embracing games. Game developers are being challenged to enlist cutting edge AI as part of their games. In this book, you will look at the journey of building capable AI using reinforcement learning algorithms and techniques. You will learn to solve complex tasks and build next-generation games using a ..