MODERN DEEP LEARNING DESIGN AND APPLICATION DEVELOPMENT : versatile tools to solve deep... learning problems

Learn how to harness modern deep-learning methods in many contexts. Packed with intuitive theory, practical implementation methods, and deep-learning case studies, this book reveals how to acquire the tools you need to design and implement like a deep-learning architect. It covers tools deep learning engineers can use in a wide range of fields, from biology to computer vision to business. With nine in-depth case studies, this book will ground you in creative, real-world deep learning thinking. You’ll begin with a structured guide to using Keras, with helpful tips and best practices for making the most of the framework. Next, you’ll learn how to train models effectively with transfer learning and self-supervised pre-training. You will then learn how to use a variety of model compressions for practical usage. Lastly, you will learn how to design successful neural network architectures and creatively reframe difficult problems into solvable ones. You’ll learn not only to understand and apply methods successfully but to think critically about it. Modern Deep Learning Design and Methods is ideal for readers looking to utilize modern, flexible, and creative deep-learning design and methods. Get ready to design and implement innovative deep-learning solutions to today’s difficult problems. What You’ll Learn Improve the performance of deep learning models by using pre-trained models, extracting rich features, and automating optimization. Compress deep learning models while maintaining performance. Reframe a wide variety of difficult problems and design effective deep learning solutions to solve them. Use the Keras framework, with some help from libraries like HyperOpt, TensorFlow, and PyTorch, to implement a wide variety of deep learning approaches. Who This Book Is For Data scientists with some familiarity with deep learning to deep learning engineers seeking structured inspiration and direction on their next project. Developers interested in harnessing modern deep learning methods to solve a variety of difficult problems. Table of Contents 5 About the Author 10 About the Technical Reviewer 11 Acknowledgments 12 Introduction 13 Chapter 1: A Deep Dive into Keras 16 Why Keras? 17 Installing and Importing Keras 18 The Simple Keras Workflow 19 Step 1: Define Architecture 19 Step 2: Compile 26 Loss Functions 28 Optimizers 36 Metrics 39 Step 3: Fit and Evaluate 42 Visualizing Model Architectures 45 Functional API 48 Translating a Sequential to a Functional Model 50 Building Nonlinear Topologies 51 Dealing with Data 56 TensorFlow Dataset from Loaded Data 57 TensorFlow Dataset from Image Files 58 Automatic Image Dataset from Directory 59 ImageDataGenerator 60 Key Points 62 Chapter 2: Pretraining Strategies and Transfer Learning 64 Developing Creative Training Structures 65 The Power of Pretraining 66 Transfer Learning Intuition 71 Self-Supervised Learning Intuition 75 Transfer Learning Practical Theory 80 Transfer Learning Models and Model Structure 81 The ImageNet Dataset 83 ResNet 85 InceptionV3 86 MobileNet 86 EfficientNet 87 Other Models 88 Changing Pretrained Model Architectures 88 Neural Network “Top” Inclusivity 93 Layer Freezing 94 Implementing Transfer Learning 96 General Implementation Structure: A Template 96 No Architecture or Weight Changes 97 Transfer Learning Without Layer Freezing 100 Transfer Learning with Layer Freezing 103 Accessing PyTorch Models 104 Implementing Simple Self-Supervised Learning 106 Case Studies 110 Transfer Learning Case Study: Adversarial Exploitation of Transfer Learning 110 Self-Supervised Learning Case Study: Predicting Rotations 117 Self-Supervised Learning Case Study: Learning Image Context and Designing Nontrivial Pretraining Tasks 123 Key Points 127 Chapter 3: The Versatility of Autoencoders 130 Autoencoder Intuition and Theory 131 The Design of Autoencoder Implementation 136 Autoencoders for Tabular Data 136 Autoencoders for Image Data 142 Image Data Shape Structure and Transformations 142 Convolutional Autoencoder Without Pooling 148 Convolutional Autoencoder Vector Bottleneck Design 152 Convolutional Autoencoder with Pooling and Padding 155 Autoencoders for Other Data Forms 160 Autoencoder Applications 160 Using Autoencoders for Denoising 161 Intuition and Theory 161 Implementation 166 Inducing Noise 166 Using Denoising Autoencoders 169 Using Autoencoders for Pretraining 171 Intuition 171 Implementation 175 Using Autoencoders for Dimensionality Reduction 176 Intuition 176 Implementation 179 Using Autoencoders for Feature Generation 179 Intuition 179 Implementation 184 Using Variational Autoencoders for Data Generation 187 Intuition 188 Implementation 196 Case Studies 203 Autoencoders for Pretraining Case Study: TabNet 204 Denoising Autoencoders Case Study: Chinese Spelling Checker 208 Variational Autoencoders Case Study: Text Generation 212 Key Points 216 Chapter 4: Model Compression for Practical Deployment 219 Introduction to Model Compression 220 Pruning 224 Pruning Theory and Intuition 224 Pruning Implementation 229 Setting Up Data and Benchmark Model 229 Creating Cost Metrics 231 Storage Size 231 Latency 233 Parameter Metrics 233 Pruning an Entire Model 234 Pruning Individual Layers 238 Pruning in Theoretical Deep Learning: The Lottery Ticket Hypothesis 240 Quantization 243 Quantization Theory and Intuition 243 Quantization Implementation 246 Quantizing an Entire Model 246 Quantizing Individual Layers 248 Weight Clustering 250 Weight Clustering Theory and Intuition 251 Weight Clustering Implementation 252 Weight Clustering on an Entire Model 252 Weight Clustering on Individual Layers 253 Collaborative Optimization 254 Sparsity Preserving Quantization 257 Cluster Preserving Quantization 259 Sparsity Preserving Clustering 260 Case Studies 262 Extreme Collaborative Optimization 262 Rethinking Quantization for Deeper Compression 265 Responsible Compression: What Do Compressed Models Forget? 268 Key Points 271 Chapter 5: Automating Model Design with Meta-optimization 273 Introduction to Meta-optimization 274 General Hyperparameter Optimization 278 Bayesian Optimization Intuition and Theory 278 Hyperopt Syntax, Concepts, and Usage 284 Hyperopt Syntax Overview: Finding the Minimum of a Simple Objective Function 285 Using Hyperopt to Optimize Training Procedure 289 Using Hyperopt to Optimize Model Architecture 292 Hyperas Syntax, Concepts, and Usage 298 Using Hyperas to Optimize Training Procedure 298 Using Hyperas to Optimize Model Architecture 302 Neural Architecture Search 303 NAS Intuition and Theory 304 Auto-Keras 311 Auto-Keras System 312 Simple NAS 316 NAS with Custom Search Space 319 NAS with Nonlinear Topology 323 Case Studies 325 NASNet 326 Progressive Neural Architecture Search 331 Efficient Neural Architecture Search 334 Key Points 337 Chapter 6: Successful Neural Network Architecture Design 340 Nonlinear and Parallel Representation 343 Residual Connections 345 Branching and Cardinality 359 Case Study: U-Net 364 Block/Cell Design 370 Sequential Cell Design 372 Nonlinear Cell Design 383 Case Study: InceptionV3 386 Neural Network Scaling 393 Input Shape Adaptable Design 395 Parametrization of Network Dimensions 401 Case Study: EfficientNet 405 Key Points 412 Chapter 7: Reframing Difficult Deep Learning Problems 414 Data Representation: DeepInsight 416 Corrupted Data: Negative Learning with Noisy Labels 427 Limited Data: Siamese Networks 440 Key Points and Epilogue 451 Index 453 Learn how to harness modern deep-learning methods in many contexts. Packed with intuitive theory, practical implementation methods, and deep-learning case studies, this book reveals how to acquire the tools you need to design and implement like a deep-learning architect. It covers tools deep learning engineers can use in a wide range of fields, from biology to computer vision to business. With nine in-depth case studies, this book will ground you in creative, real-world deep learning thinking. You'll begin with a structured guide to using Keras, with helpful tips and best practices for making the most of the framework. Next, you'll learn how to train models effectively with transfer learning and self-supervised pre-training. You will then learn how to use a variety of model compressions for practical usage. Lastly, you will learn how to design successful neural network architectures and creatively reframe difficult problems into solvable ones. You'll learn not only to understand and apply methods successfully but to think critically about it. Modern Deep Learning Design and Methods is ideal for readers looking to utilize modern, flexible, and creative deep-learning design and methods. Get ready to design and implement innovative deep-learning solutions to today's difficult problems. You will: Improve the performance of deep learning models by using pre-trained models, extracting rich features, and automating optimization. Compress deep learning models while maintaining performance. Reframe a wide variety of difficult problems and design effective deep learning solutions to solve them. Use the Keras framework, with some help from libraries like HyperOpt, TensorFlow, and PyTorch, to implement a wide variety of deep learning approaches