This open access book presents the first comprehensive overview of general methods in Automated Machine Learning (AutoML), collects descriptions of existing systems based on these methods, and discusses the first series of international challenges of AutoML systems. The recent success of commercial ML applications and the rapid growth of the field has created a high demand for off-the-shelf ML methods that can be used easily and without expert knowledge. However, many of the recent machine learning successes crucially rely on human experts, who manually select appropriate ML architectures (deep learning architectures or more traditional ML workflows) and their hyperparameters. To overcome this problem, the field of AutoML targets a progressive automation of machine learning, based on principles from optimization and machine learning itself. This book serves as a point of entry into this quickly-developing field for researchers and advanced students alike, as well as providing a reference for practitioners aiming to use AutoML in their work. Foreword Preface Acknowledgments Contents Part I AutoML Methods 1 Hyperparameter Optimization 1.1 Introduction 1.2 Problem Statement 1.2.1 Alternatives to Optimization: Ensembling and Marginalization 1.2.2 Optimizing for Multiple Objectives 1.3 Blackbox Hyperparameter Optimization 1.3.1 Model-Free Blackbox Optimization Methods 1.3.2 Bayesian Optimization 1.3.2.1 Bayesian Optimization in a Nutshell 1.3.2.2 Surrogate Models 1.3.2.3 Configuration Space Description 1.3.2.4 Constrained Bayesian Optimization 1.4 Multi-fidelity Optimization 1.4.1 Learning Curve-Based Prediction for Early Stopping 1.4.2 Bandit-Based Algorithm Selection Methods 1.4.3 Adaptive Choices of Fidelities 1.5 Applications to AutoML 1.6 Open Problems and Future Research Directions 1.6.1 Benchmarks and Comparability 1.6.2 Gradient-Based Optimization 1.6.3 Scalability 1.6.4 Overfitting and Generalization 1.6.5 Arbitrary-Size Pipeline Construction Bibliography 2 Meta-Learning 2.1 Introduction 2.2 Learning from Model Evaluations 2.2.1 Task-Independent Recommendations 2.2.2 Configuration Space Design 2.2.3 Configuration Transfer 2.2.3.1 Relative Landmarks 2.2.3.2 Surrogate Models 2.2.3.3 Warm-Started Multi-task Learning 2.2.3.4 Other Techniques 2.2.4 Learning Curves 2.3 Learning from Task Properties 2.3.1 Meta-Features 2.3.2 Learning Meta-Features 2.3.3 Warm-Starting Optimization from Similar Tasks 2.3.4 Meta-Models 2.3.4.1 Ranking 2.3.4.2 Performance Prediction 2.3.5 Pipeline Synthesis 2.3.6 To Tune or Not to Tune? 2.4 Learning from Prior Models 2.4.1 Transfer Learning 2.4.2 Meta-Learning in Neural Networks 2.4.3 Few-Shot Learning 2.4.4 Beyond Supervised Learning 2.5 Conclusion Bibliography 3 Neural Architecture Search 3.1 Introduction 3.2 Search Space 3.3 Search Strategy 3.4 Performance Estimation Strategy 3.5 Future Directions Bibliography Part II AutoML Systems 4 Auto-WEKA: Automatic Model Selection and Hyperparameter Optimization in WEKA 4.1 Introduction 4.2 Preliminaries 4.2.1 Model Selection 4.2.2 Hyperparameter Optimization 4.3 CASH 4.3.1 Sequential Model-Based Algorithm Configuration (SMAC) 4.4 Auto-WEKA 4.5 Experimental Evaluation 4.5.1 Baseline Methods 4.5.2 Results for Cross-Validation Performance 4.5.3 Results for Test Performance 4.6 Conclusion 4.6.1 Community Adoption Bibliography 5 Hyperopt-Sklearn 5.1 Introduction 5.2 Background: Hyperopt for Optimization 5.3 Scikit-Learn Model Selection as a Search Problem 5.4 Example Usage 5.5 Experiments 5.6 Discussion and Future Work 5.7 Conclusions Bibliography 6 Auto-sklearn: Efficient and Robust Automated MachineLearning 6.1 Introduction 6.2 AutoML as a CASH Problem 6.3 New Methods for Increasing Efficiency and Robustness of AutoML 6.3.1 Meta-learning for Finding Good Instantiations of Machine Learning Frameworks 6.3.2 Automated Ensemble Construction of Models Evaluated During Optimization 6.4 A Practical Automated Machine Learning System 6.5 Comparing Auto-sklearn to Auto-WEKA and Hyperopt-Sklearn 6.6 Evaluation of the Proposed AutoML Improvements 6.7 Detailed Analysis of Auto-sklearn Components 6.8 Discussion and Conclusion 6.8.1 Discussion 6.8.2 Usage 6.8.3 Extensions in PoSH Auto-sklearn 6.8.4 Conclusion and Future Work Bibliography 7 Towards Automatically-Tuned Deep Neural Networks 7.1 Introduction 7.2 Auto-Net 1.0 7.3 Auto-Net 2.0 7.4 Experiments 7.4.1 Baseline Evaluation of Auto-Net 1.0 and Auto-sklearn 7.4.2 Results for AutoML Competition Datasets 7.4.3 Comparing AutoNet 1.0 and 2.0 7.5 Conclusion Bibliography 8 TPOT: A Tree-Based Pipeline Optimization Toolfor Automating Machine Learning 8.1 Introduction 8.2 Methods 8.2.1 Machine Learning Pipeline Operators 8.2.2 Constructing Tree-Based Pipelines 8.2.3 Optimizing Tree-Based Pipelines 8.2.4 Benchmark Data 8.3 Results 8.4 Conclusions and Future Work Bibliography 9 The Automatic Statistician 9.1 Introduction 9.2 Basic Anatomy of an Automatic Statistician 9.2.1 Related Work 9.3 An Automatic Statistician for Time Series Data 9.3.1 The Grammar over Kernels 9.3.2 The Search and Evaluation Procedure 9.3.3 Generating Descriptions in Natural Language 9.3.4 Comparison with Humans 9.4 Other Automatic Statistician Systems 9.4.1 Core Components 9.4.2 Design Challenges 9.4.2.1 User Interaction 9.4.2.2 Missing and Messy Data 9.4.2.3 Resource Allocation 9.5 Conclusion Bibliography Part III AutoML Challenges 10 Analysis of the AutoML Challenge Series 2015–2018 10.1 Introduction 10.2 Problem Formalization and Overview 10.2.1 Scope of the Problem 10.2.2 Full Model Selection 10.2.3 Optimization of Hyper-parameters 10.2.4 Strategies of Model Search 10.3 Data 10.4 Challenge Protocol 10.4.1 Time Budget and Computational Resources 10.4.2 Scoring Metrics 10.4.3 Rounds and Phases in the 2015/2016 Challenge 10.4.4 Phases in the 2018 Challenge 10.5 Results 10.5.1 Scores Obtained in the 2015/2016 Challenge 10.5.2 Scores Obtained in the 2018 Challenge 10.5.3 Difficulty of Datasets/Tasks 10.5.4 Hyper-parameter Optimization 10.5.5 Meta-learning 10.5.6 Methods Used in the Challenges 10.6 Discussion 10.7 Conclusion Bibliography
