Hyperparameter Optimization in Machine Learning : Make Your Machine Learning and Deep Learning Models More Efficient

Dive into hyperparameter tuning of machine learning models and focus on what hyperparameters are and how they work. This book discusses different techniques of hyperparameters tuning, from the basics to advanced methods. This is a step-by-step guide to hyperparameter optimization, starting with what hyperparameters are and how they affect different aspects of machine learning models. It then goes through some basic (brute force) algorithms of hyperparameter optimization. Further, the author addresses the problem of time and memory constraints, using distributed optimization methods. Next you'll discuss Bayesian optimization for hyperparameter search, which learns from its previous history. The book discusses different frameworks, such as Hyperopt and Optuna, which implements sequential model-based global optimization (SMBO) algorithms. During these discussions, you'll focus on different aspects such as creation of search spaces and distributed optimization of these libraries. Hyperparameter Optimization in Machine Learning creates an understanding of how these algorithms work and how you can use them in real-life data science problems. The final chapter summaries the role of hyperparameter optimization in automated machine learning and ends with a tutorial to create your own AutoML script. Hyperparameter optimization is tedious task, so sit back and let these algorithms do your work. What You Will Learn Discover how changes in hyperparameters affect the model's performance. Apply different hyperparameter tuning algorithms to data science problems Work with Bayesian optimization methods to create efficient machine learning and deep learning models Distribute hyperparameter optimization using a cluster of machines Approach automated machine learning using hyperparameter optimization Who This Book Is For Professionals and students working with machine learning. Table of Contents 5 About the Author 8 About the Technical Reviewer 9 Acknowledgments 10 Foreword 1 11 Foreword 2 13 Introduction 14 Chapter 1: Introduction to Hyperparameters 15 Introduction to Machine Learning 16 Understanding Hyperparameters 18 The Need for Hyperparameter Optimization 22 Algorithms and Their Hyperparameters 25 K-Nearest Neighbor 25 Support Vector Machine 27 Decision Tree 30 Neural Networks 33 Distribution of Possible Hyperparameter Values 35 Discrete Variables 36 Continuous Variables 38 Probabilistic Distributions 38 Uniform Distribution 39 Gaussian Distribution 41 Exponential Distribution 43 Chapter 2: Hyperparameter Optimization Using Scikit-Learn 45 Changing Hyperparameters 45 Grid Search 47 Random Search 52 Parallel Hyperparameter Optimization 56 Chapter 3: Solving Time and Memory Constraints 66 Dask 67 Dask Distributed 68 Parallel Collections 70 Dynamic Task Scheduling 73 Hyperparameter Optimization with Dask 76 Dask Random Search and Grid Search 77 Incremental Search 79 Successive Halving Search 81 Hyperband Search 82 Distributing Deep Learning Models 84 PyTorch Distributed 85 Horovod 90 Chapter 4: Bayesian Optimization 94 Sequential Model-Based Global Optimization 95 Tree-Structured Parzen Estimator 99 Hyperopt 102 Search Space 105 Parallelizing Trials in TPE 113 Hyperopt-Sklearn 115 Hyperas 117 Chapter 5: Optuna and AutoML 122 Optuna 122 Search Space 125 Underlying Algorithms 126 Visualization 127 Distributed Optimization 127 Automated Machine Learning 132 Building Your Own AutoML Module 132 TPOT 139 Appendix I 143 Data Cleaning and Preprocessing 143 Dealing with Nonnumerical Columns 143 Label Encoding 144 One-Hot Encoding 144 Missing Values 145 Drop the Rows 146 Mean/Median or Most Frequent/Constant 146 Imputation Using Regression or Classification 146 Multivariate Imputation by Chained Equations1 147 Outlier Detection 148 Z-score 149 Density-Based Spatial Clustering of Applications with Noise 149 Feature Selection 150 F-Test 151 Mutual Information Test 151 Recursive Feature Selection 152 Applying the Techniques 152 Applying Machine Learning Algorithms 157 Model Evaluation Methods 158 Appendix II: Neural Networks: A Brief Introduction to PyTorch and Keras API 164 Index 174