Automated Machine Learning: Methods, Systems, Challenges (The Springer Series on Challenges in Machine Learning)

This open access book presents the first comprehensive overview of general methods in Automated Machine Learning (AutoML), collects descriptions of existing systems based on these methods, and discusses the first series of international challenges of AutoML systems. The recent success of commercial ML applications and the rapid growth of the field has created a high demand for off-the-shelf ML methods that can be used easily and without expert knowledge. However, many of the recent machine learning successes crucially rely on human experts, who manually select appropriate ML architectures (deep learning architectures or more traditional ML workflows) and their hyperparameters. To overcome this problem, the field of AutoML targets a progressive automation of machine learning, based on principles from optimization and machine learning itself. This book serves as a point of entry into this quickly-developing field for researchers and advanced students alike, as well as providing a reference for practitioners aiming to use AutoML in their work. Foreword 7 Preface 9 Acknowledgments 11 Contents 12 Part I AutoML Methods 14 1 Hyperparameter Optimization 15 1.1 Introduction 15 1.2 Problem Statement 17 1.2.1 Alternatives to Optimization: Ensembling and Marginalization 18 1.2.2 Optimizing for Multiple Objectives 18 1.3 Blackbox Hyperparameter Optimization 19 1.3.1 Model-Free Blackbox Optimization Methods 19 1.3.2 Bayesian Optimization 21 1.3.2.1 Bayesian Optimization in a Nutshell 21 1.3.2.2 Surrogate Models 22 1.3.2.3 Configuration Space Description 24 1.3.2.4 Constrained Bayesian Optimization 25 1.4 Multi-fidelity Optimization 26 1.4.1 Learning Curve-Based Prediction for Early Stopping 26 1.4.2 Bandit-Based Algorithm Selection Methods 27 1.4.3 Adaptive Choices of Fidelities 29 1.5 Applications to AutoML 30 1.6 Open Problems and Future Research Directions 33 1.6.1 Benchmarks and Comparability 33 1.6.2 Gradient-Based Optimization 34 1.6.3 Scalability 35 1.6.4 Overfitting and Generalization 35 1.6.5 Arbitrary-Size Pipeline Construction 36 Bibliography 37 2 Meta-Learning 46 2.1 Introduction 46 2.2 Learning from Model Evaluations 47 2.2.1 Task-Independent Recommendations 48 2.2.2 Configuration Space Design 48 2.2.3 Configuration Transfer 49 2.2.3.1 Relative Landmarks 49 2.2.3.2 Surrogate Models 50 2.2.3.3 Warm-Started Multi-task Learning 50 2.2.3.4 Other Techniques 51 2.2.4 Learning Curves 52 2.3 Learning from Task Properties 52 2.3.1 Meta-Features 53 2.3.2 Learning Meta-Features 53 2.3.3 Warm-Starting Optimization from Similar Tasks 55 2.3.4 Meta-Models 57 2.3.4.1 Ranking 57 2.3.4.2 Performance Prediction 57 2.3.5 Pipeline Synthesis 58 2.3.6 To Tune or Not to Tune? 59 2.4 Learning from Prior Models 59 2.4.1 Transfer Learning 59 2.4.2 Meta-Learning in Neural Networks 60 2.4.3 Few-Shot Learning 61 2.4.4 Beyond Supervised Learning 62 2.5 Conclusion 63 Bibliography 64 3 Neural Architecture Search 73 3.1 Introduction 73 3.2 Search Space 74 3.3 Search Strategy 77 3.4 Performance Estimation Strategy 80 3.5 Future Directions 82 Bibliography 84 Part II AutoML Systems 88 4 Auto-WEKA: Automatic Model Selection and Hyperparameter Optimization in WEKA 89 4.1 Introduction 89 4.2 Preliminaries 91 4.2.1 Model Selection 91 4.2.2 Hyperparameter Optimization 92 4.3 CASH 92 4.3.1 Sequential Model-Based Algorithm Configuration (SMAC) 94 4.4 Auto-WEKA 95 4.5 Experimental Evaluation 96 4.5.1 Baseline Methods 97 4.5.2 Results for Cross-Validation Performance 98 4.5.3 Results for Test Performance 100 4.6 Conclusion 101 4.6.1 Community Adoption 101 Bibliography 101 5 Hyperopt-Sklearn 104 5.1 Introduction 104 5.2 Background: Hyperopt for Optimization 105 5.3 Scikit-Learn Model Selection as a Search Problem 107 5.4 Example Usage 108 5.5 Experiments 112 5.6 Discussion and Future Work 113 5.7 Conclusions 117 Bibliography 117 6 Auto-sklearn: Efficient and Robust Automated MachineLearning 119 6.1 Introduction 119 6.2 AutoML as a CASH Problem 121 6.3 New Methods for Increasing Efficiency and Robustness of AutoML 122 6.3.1 Meta-learning for Finding Good Instantiations of Machine Learning Frameworks 122 6.3.2 Automated Ensemble Construction of Models Evaluated During Optimization 124 6.4 A Practical Automated Machine Learning System 125 6.5 Comparing Auto-sklearn to Auto-WEKA and Hyperopt-Sklearn 128 6.6 Evaluation of the Proposed AutoML Improvements 128 6.7 Detailed Analysis of Auto-sklearn Components 131 6.8 Discussion and Conclusion 132 6.8.1 Discussion 132 6.8.2 Usage 136 6.8.3 Extensions in PoSH Auto-sklearn 136 6.8.4 Conclusion and Future Work 137 Bibliography 138 7 Towards Automatically-Tuned Deep Neural Networks 141 7.1 Introduction 141 7.2 Auto-Net 1.0 143 7.3 Auto-Net 2.0 145 7.4 Experiments 149 7.4.1 Baseline Evaluation of Auto-Net 1.0 and Auto-sklearn 149 7.4.2 Results for AutoML Competition Datasets 149 7.4.3 Comparing AutoNet 1.0 and 2.0 151 7.5 Conclusion 152 Bibliography 153 8 TPOT: A Tree-Based Pipeline Optimization Toolfor Automating Machine Learning 156 8.1 Introduction 156 8.2 Methods 157 8.2.1 Machine Learning Pipeline Operators 157 8.2.2 Constructing Tree-Based Pipelines 158 8.2.3 Optimizing Tree-Based Pipelines 159 8.2.4 Benchmark Data 159 8.3 Results 160 8.4 Conclusions and Future Work 163 Bibliography 164 9 The Automatic Statistician 166 9.1 Introduction 166 9.2 Basic Anatomy of an Automatic Statistician 168 9.2.1 Related Work 169 9.3 An Automatic Statistician for Time Series Data 169 9.3.1 The Grammar over Kernels 169 9.3.2 The Search and Evaluation Procedure 170 9.3.3 Generating Descriptions in Natural Language 171 9.3.4 Comparison with Humans 173 9.4 Other Automatic Statistician Systems 174 9.4.1 Core Components 174 9.4.2 Design Challenges 175 9.4.2.1 User Interaction 175 9.4.2.2 Missing and Messy Data 175 9.4.2.3 Resource Allocation 175 9.5 Conclusion 176 Bibliography 176 Part III AutoML Challenges 179 10 Analysis of the AutoML Challenge Series 2015–2018 180 10.1 Introduction 181 10.2 Problem Formalization and Overview 184 10.2.1 Scope of the Problem 184 10.2.2 Full Model Selection 185 10.2.3 Optimization of Hyper-parameters 186 10.2.4 Strategies of Model Search 187 10.3 Data 191 10.4 Challenge Protocol 193 10.4.1 Time Budget and Computational Resources 194 10.4.2 Scoring Metrics 195 10.4.3 Rounds and Phases in the 2015/2016 Challenge 197 10.4.4 Phases in the 2018 Challenge 198 10.5 Results 199 10.5.1 Scores Obtained in the 2015/2016 Challenge 199 10.5.2 Scores Obtained in the 2018 Challenge 201 10.5.3 Difficulty of Datasets/Tasks 202 10.5.4 Hyper-parameter Optimization 208 10.5.5 Meta-learning 209 10.5.6 Methods Used in the Challenges 211 10.6 Discussion 215 10.7 Conclusion 216 Bibliography 218 Front Matter ....Pages i-xiv Front Matter ....Pages 1-1 Hyperparameter Optimization (Matthias Feurer, Frank Hutter)....Pages 3-33 Meta-Learning (Joaquin Vanschoren)....Pages 35-61 Neural Architecture Search (Thomas Elsken, Jan Hendrik Metzen, Frank Hutter)....Pages 63-77 Front Matter ....Pages 79-79 Auto-WEKA: Automatic Model Selection and Hyperparameter Optimization in WEKA (Lars Kotthoff, Chris Thornton, Holger H. Hoos, Frank Hutter, Kevin Leyton-Brown)....Pages 81-95 Hyperopt-Sklearn (Brent Komer, James Bergstra, Chris Eliasmith)....Pages 97-111 Auto-sklearn: Efficient and Robust Automated Machine Learning (Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Tobias Springenberg, Manuel Blum, Frank Hutter)....Pages 113-134 Towards Automatically-Tuned Deep Neural Networks (Hector Mendoza, Aaron Klein, Matthias Feurer, Jost Tobias Springenberg, Matthias Urban, Michael Burkart et al.)....Pages 135-149 TPOT: A Tree-Based Pipeline Optimization Tool for Automating Machine Learning (Randal S. Olson, Jason H. Moore)....Pages 151-160 The Automatic Statistician (Christian Steinruecken, Emma Smith, David Janz, James Lloyd, Zoubin Ghahramani)....Pages 161-173 Front Matter ....Pages 175-175 Analysis of the AutoML Challenge Series 2015–2018 (Isabelle Guyon, Lisheng Sun-Hosoya, Marc Boullé, Hugo Jair Escalante, Sergio Escalera, Zhengying Liu et al.)....Pages 177-219 Correction to: Neural Architecture Search (Thomas Elsken, Jan Hendrik Metzen, Frank Hutter)....Pages C1-C1